[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:13 dist-test-slave-sfvt krb5kdc[1328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:13 dist-test-slave-sfvt krb5kdc[1328](info): set up 2 sockets
Sep 26 06:57:13 dist-test-slave-sfvt krb5kdc[1328](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 06:57:14.542945 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:14 dist-test-slave-sfvt krb5kdc[1328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:14.657959 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:34595
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:34595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:14.736943 1341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:14.737133 1341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:14.737156 1341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:14.738247 1341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:14.738284 1341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:14.738298 1341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:14.738310 1341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:14.738322 1341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:14.739728 1341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:34595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:34595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:14.739913 1341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:14.740104 1341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:57:14.745286 1341 server_base.cc:1034] running on GCE node
W20260926 06:57:14.745209 1346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:14.745261 1347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:14.745261 1349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:14.745774 1341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:14.746047 1341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:14.747213 1341 hybrid_clock.cc:648] HybridClock initialized: now 1790405834747199 us; error 41 us; skew 500 ppm
Sep 26 06:57:14 dist-test-slave-sfvt krb5kdc[1328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405834, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:14.752735 1341 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:14.753194 1341 webserver.cc:460] Webserver started at http://127.1.74.126:42257/ using document root <none> and password file <none>
I20260926 06:57:14.753403 1341 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:14.753449 1341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:14.753551 1341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:14.754632 1341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "cc287eee3a0e4c17a174eee58b841c58"
format_stamp: "Formatted at 2026-09-26 06:57:14 on dist-test-slave-sfvt"
server_key: "b843e0582e6ef1aa7f0ce685018e5798"
server_key_iv: "da8ff5991177cf7456c147d63592c8c1"
server_key_version: "encryptionkey@0"
I20260926 06:57:14.754961 1341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "cc287eee3a0e4c17a174eee58b841c58"
format_stamp: "Formatted at 2026-09-26 06:57:14 on dist-test-slave-sfvt"
server_key: "b843e0582e6ef1aa7f0ce685018e5798"
server_key_iv: "da8ff5991177cf7456c147d63592c8c1"
server_key_version: "encryptionkey@0"
I20260926 06:57:14.756412 1341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:14.757124 1356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:14.757300 1341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:14.757380 1341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "cc287eee3a0e4c17a174eee58b841c58"
format_stamp: "Formatted at 2026-09-26 06:57:14 on dist-test-slave-sfvt"
server_key: "b843e0582e6ef1aa7f0ce685018e5798"
server_key_iv: "da8ff5991177cf7456c147d63592c8c1"
server_key_version: "encryptionkey@0"
I20260926 06:57:14.757431 1341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:14.777812 1341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:14.779704 1341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:14.779832 1341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:14.783919 1341 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:34595
I20260926 06:57:14.783996 1408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:34595 every 8 connection(s)
I20260926 06:57:14.784317 1341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:14.785174 1409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:14.787451 1409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58: Bootstrap starting.
I20260926 06:57:14.788025 1409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:14.788308 1409 log.cc:824] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:14.788975 1409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58: No bootstrap required, opened a new log
I20260926 06:57:14.790887 1409 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc287eee3a0e4c17a174eee58b841c58" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34595 } }
I20260926 06:57:14.791026 1409 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:14.791075 1409 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc287eee3a0e4c17a174eee58b841c58, State: Initialized, Role: FOLLOWER
I20260926 06:57:14.791244 1409 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc287eee3a0e4c17a174eee58b841c58" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34595 } }
I20260926 06:57:14.791344 1409 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:14.791385 1409 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:14.791442 1409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:14.791971 1409 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc287eee3a0e4c17a174eee58b841c58" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34595 } }
I20260926 06:57:14.792057 1409 leader_election.cc:302] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc287eee3a0e4c17a174eee58b841c58; no voters:
I20260926 06:57:14.792243 1409 leader_election.cc:288] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:14.792272 1412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:14.792425 1412 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [term 1 LEADER]: Becoming Leader. State: Replica: cc287eee3a0e4c17a174eee58b841c58, State: Running, Role: LEADER
I20260926 06:57:14.792505 1412 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc287eee3a0e4c17a174eee58b841c58" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34595 } }
I20260926 06:57:14.792547 1409 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:14.792671 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 1341
I20260926 06:57:14.792768 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:14.792785 1413 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc287eee3a0e4c17a174eee58b841c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc287eee3a0e4c17a174eee58b841c58" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34595 } } }
I20260926 06:57:14.792858 1413 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:14.792940 1414 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc287eee3a0e4c17a174eee58b841c58. Latest consensus state: current_term: 1 leader_uuid: "cc287eee3a0e4c17a174eee58b841c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc287eee3a0e4c17a174eee58b841c58" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34595 } } }
I20260926 06:57:14.792994 1414 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:14.793232 1416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:14.793926 1321 external_mini_cluster.cc:1428] Setting key 9269ca720444db805526ccaf2ba47db2
I20260926 06:57:14.793963 1416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:14.795639 1416 catalog_manager.cc:1347] Generated new cluster ID: df6d59553503469fb0053b32f29ab060
I20260926 06:57:14.795691 1416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:57:14 dist-test-slave-sfvt krb5kdc[1328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:14.808089 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:14.794764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.1.74.126:34595)
0926 06:57:14.795309 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:14.795317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:14.795353 (+ 36us) server_negotiation.cc:407] Connection header received
0926 06:57:14.797043 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:14.797049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:14.797100 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:14.797231 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:14.798631 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:14.799662 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:14.800689 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:14.801103 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:14.801231 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:14.804589 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:14.804593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:14.804594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:14.804611 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:14.806239 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:14.806670 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:14.806673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:14.806673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:14.806758 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:14.807008 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:14.807009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:14.807010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:14.807238 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:14.807413 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:14.807684 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 06:57:14.807815 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":74,"threads_started":1}
I20260926 06:57:14.809401 1416 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:14.809885 1416 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:14.814226 1416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc287eee3a0e4c17a174eee58b841c58: Generated new TSK 0
I20260926 06:57:14.814393 1416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:14.843586 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34595
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:14.951589 1435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:14.951807 1435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:14.951855 1435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:14.953069 1435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:14.953130 1435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:14.953176 1435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:14.954617 1435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34595
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:14.954882 1435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:14.955104 1435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:14.961323 1440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:14.961504 1435 server_base.cc:1034] running on GCE node
W20260926 06:57:14.961331 1441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:14.961295 1443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:14.961810 1435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:14.962055 1435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:14.963209 1435 hybrid_clock.cc:648] HybridClock initialized: now 1790405834963197 us; error 28 us; skew 500 ppm
Sep 26 06:57:14 dist-test-slave-sfvt krb5kdc[1328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405834, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:14.965746 1435 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:14.966193 1435 webserver.cc:460] Webserver started at http://127.1.74.65:45357/ using document root <none> and password file <none>
I20260926 06:57:14.966408 1435 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:14.966471 1435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:14.966612 1435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:14.967720 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "9a2e0a45180a4849b55003843183e040"
format_stamp: "Formatted at 2026-09-26 06:57:14 on dist-test-slave-sfvt"
server_key: "7acd5f468338670cc51714e9f5a13827"
server_key_iv: "4583eae48446f2796739aa9cacdaa4d9"
server_key_version: "encryptionkey@0"
I20260926 06:57:14.968091 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "9a2e0a45180a4849b55003843183e040"
format_stamp: "Formatted at 2026-09-26 06:57:14 on dist-test-slave-sfvt"
server_key: "7acd5f468338670cc51714e9f5a13827"
server_key_iv: "4583eae48446f2796739aa9cacdaa4d9"
server_key_version: "encryptionkey@0"
I20260926 06:57:14.969372 1435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:57:14.970080 1450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:14.970252 1435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:14.970347 1435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "9a2e0a45180a4849b55003843183e040"
format_stamp: "Formatted at 2026-09-26 06:57:14 on dist-test-slave-sfvt"
server_key: "7acd5f468338670cc51714e9f5a13827"
server_key_iv: "4583eae48446f2796739aa9cacdaa4d9"
server_key_version: "encryptionkey@0"
I20260926 06:57:14.970434 1435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:14.981642 1435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:14.983644 1435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:14.983803 1435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:14.984426 1435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:14.984740 1435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:14.984789 1435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:14.984829 1435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:14.984867 1435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:14.991504 1435 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:35217
I20260926 06:57:14.991569 1563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:35217 every 8 connection(s)
I20260926 06:57:14.991942 1435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:57:14 dist-test-slave-sfvt krb5kdc[1328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405834, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:15.000749 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:14.992682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:46733 (local address 127.1.74.126:34595)
0926 06:57:14.992881 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:14.992883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:14.992920 (+ 37us) server_negotiation.cc:407] Connection header received
0926 06:57:14.993582 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:14.993584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:14.993595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:14.993661 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:14.994659 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:14.995152 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:14.996128 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:14.996318 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:14.996370 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:14.998402 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:14.998403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:14.998403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:14.998409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:14.999662 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.000010 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.000011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.000011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.000059 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.000252 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.000253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.000253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.000347 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:15.000394 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:15.000621 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 06:57:15.000687 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260926 06:57:15.000953 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 1435
I20260926 06:57:15.000972 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:14.992668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:34595 (local address 127.1.74.65:46733)
0926 06:57:14.992822 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:14.992857 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:14.993400 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:14.993719 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:14.993723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:14.994379 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:14.994581 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:14.994586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:14.995226 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:14.995227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:14.996056 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:14.996058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:14.996384 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:14.996385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:14.996480 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:14.997015 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:14.997023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:14.998327 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:14.999734 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:14.999740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:14.999742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:14.999953 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:15.000111 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:15.000112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:15.000112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:15.000203 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:15.000394 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:15.000398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:15.000506 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 06:57:15.000596 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 06:57:15.000713 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":78,"threads_started":1}
I20260926 06:57:15.001057 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:15.001231 1321 external_mini_cluster.cc:1428] Setting key 50e7756ca9124d26ef3d3ec3df8b120d
I20260926 06:57:15.001435 1564 heartbeater.cc:347] Connected to a master server at 127.1.74.126:34595
I20260926 06:57:15.001544 1564 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:15.001698 1564 heartbeater.cc:511] Master 127.1.74.126:34595 requested a full tablet report, sending...
I20260926 06:57:15.002390 1373 ts_manager.cc:194] Registered new tserver with Master: 9a2e0a45180a4849b55003843183e040 (127.1.74.65:35217)
I20260926 06:57:15.003011 1373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:46733
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:15.025851 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34595
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:15.108258 1569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:15.108484 1569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:15.108511 1569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:15.109629 1569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:15.109665 1569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:15.109692 1569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:15.111056 1569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34595
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:15.111236 1569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:15.111508 1569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:15.116434 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:15.116436 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:15.116523 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:15.116664 1569 server_base.cc:1034] running on GCE node
I20260926 06:57:15.116916 1569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:15.117125 1569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:15.118314 1569 hybrid_clock.cc:648] HybridClock initialized: now 1790405835118306 us; error 37 us; skew 500 ppm
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405835, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:15.120879 1569 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:15.121356 1569 webserver.cc:460] Webserver started at http://127.1.74.66:34815/ using document root <none> and password file <none>
I20260926 06:57:15.121583 1569 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:15.121628 1569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:15.121795 1569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:15.122877 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "2d5a7783e5134bdfa02a4772f5680744"
format_stamp: "Formatted at 2026-09-26 06:57:15 on dist-test-slave-sfvt"
server_key: "3a82b326d365b3854f387f298172e644"
server_key_iv: "83228ddbf78824a6525213e2d6eda92c"
server_key_version: "encryptionkey@0"
I20260926 06:57:15.123246 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "2d5a7783e5134bdfa02a4772f5680744"
format_stamp: "Formatted at 2026-09-26 06:57:15 on dist-test-slave-sfvt"
server_key: "3a82b326d365b3854f387f298172e644"
server_key_iv: "83228ddbf78824a6525213e2d6eda92c"
server_key_version: "encryptionkey@0"
I20260926 06:57:15.124653 1569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:15.125375 1584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:15.125547 1569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:15.125641 1569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "2d5a7783e5134bdfa02a4772f5680744"
format_stamp: "Formatted at 2026-09-26 06:57:15 on dist-test-slave-sfvt"
server_key: "3a82b326d365b3854f387f298172e644"
server_key_iv: "83228ddbf78824a6525213e2d6eda92c"
server_key_version: "encryptionkey@0"
I20260926 06:57:15.125721 1569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:15.142076 1569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:15.143996 1569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:15.144150 1569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:15.144375 1569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:15.144666 1569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:15.144716 1569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:15.144757 1569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:15.144791 1569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:15.151134 1569 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:35883
I20260926 06:57:15.151178 1697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:35883 every 8 connection(s)
I20260926 06:57:15.151501 1569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405835, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:15.160185 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:15.152172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:39251 (local address 127.1.74.126:34595)
0926 06:57:15.152256 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:15.152259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:15.152431 (+ 172us) server_negotiation.cc:407] Connection header received
0926 06:57:15.152939 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:15.152940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:15.152950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:15.153021 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:15.153959 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.154453 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.155498 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.155683 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.155743 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:15.157770 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:15.157771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:15.157772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:15.157778 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:15.159014 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.159434 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.159435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.159435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.159490 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.159689 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.159690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.159690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.159791 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:15.159843 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:15.160055 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 06:57:15.160121 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:57:15.160360 1700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:15.152145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:34595 (local address 127.1.74.66:39251)
0926 06:57:15.152316 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:15.152352 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:15.152818 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:15.153092 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:15.153095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:15.153666 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:15.153854 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:15.153858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.154544 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.154544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:15.155408 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:15.155410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.155754 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.155755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:15.155850 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:15.156286 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:15.156294 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:15.157688 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:15.159088 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:15.159093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:15.159095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:15.159363 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:15.159544 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:15.159544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:15.159544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:15.159639 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:15.159844 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:15.159848 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:15.159955 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 06:57:15.160032 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:57:15.160155 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260926 06:57:15.160799 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 1569
I20260926 06:57:15.160856 1698 heartbeater.cc:347] Connected to a master server at 127.1.74.126:34595
I20260926 06:57:15.160914 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:15.160943 1698 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:15.161095 1321 external_mini_cluster.cc:1428] Setting key 10a8990cf94f99af65125503ab58cc6e
I20260926 06:57:15.161134 1698 heartbeater.cc:511] Master 127.1.74.126:34595 requested a full tablet report, sending...
I20260926 06:57:15.161499 1372 ts_manager.cc:194] Registered new tserver with Master: 2d5a7783e5134bdfa02a4772f5680744 (127.1.74.66:35883)
I20260926 06:57:15.161866 1372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:39251
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:15.186960 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34595
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:15.268594 1703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:15.268790 1703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:15.268813 1703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:15.269934 1703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:15.269973 1703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:15.270000 1703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:15.271598 1703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34595
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:15.271821 1703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:15.272032 1703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:15.276940 1709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:15.276952 1708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:15.276952 1711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:15.277122 1703 server_base.cc:1034] running on GCE node
I20260926 06:57:15.277352 1703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:15.277581 1703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:15.278728 1703 hybrid_clock.cc:648] HybridClock initialized: now 1790405835278713 us; error 26 us; skew 500 ppm
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405835, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:15.281589 1703 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:15.281957 1703 webserver.cc:460] Webserver started at http://127.1.74.67:40329/ using document root <none> and password file <none>
I20260926 06:57:15.282150 1703 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:15.282212 1703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:15.282305 1703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:15.283388 1703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "41fe8b4e71bb4bdc942983ae068401c7"
format_stamp: "Formatted at 2026-09-26 06:57:15 on dist-test-slave-sfvt"
server_key: "f086e3934cbe038ee7514744d3329989"
server_key_iv: "85bcf29d0e806954a7a1d992b69729d5"
server_key_version: "encryptionkey@0"
I20260926 06:57:15.283738 1703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "41fe8b4e71bb4bdc942983ae068401c7"
format_stamp: "Formatted at 2026-09-26 06:57:15 on dist-test-slave-sfvt"
server_key: "f086e3934cbe038ee7514744d3329989"
server_key_iv: "85bcf29d0e806954a7a1d992b69729d5"
server_key_version: "encryptionkey@0"
I20260926 06:57:15.284945 1703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:15.285624 1718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:15.285781 1703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:15.285867 1703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "41fe8b4e71bb4bdc942983ae068401c7"
format_stamp: "Formatted at 2026-09-26 06:57:15 on dist-test-slave-sfvt"
server_key: "f086e3934cbe038ee7514744d3329989"
server_key_iv: "85bcf29d0e806954a7a1d992b69729d5"
server_key_version: "encryptionkey@0"
I20260926 06:57:15.285940 1703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:15.302215 1703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:15.304149 1703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:15.304311 1703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:15.304553 1703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:15.304852 1703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:15.304901 1703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:15.304940 1703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:15.304973 1703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:15.311896 1703 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:36923
I20260926 06:57:15.311935 1831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:36923 every 8 connection(s)
I20260926 06:57:15.312256 1703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405835, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:15.321746 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:15.313027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:37419 (local address 127.1.74.126:34595)
0926 06:57:15.313099 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:15.313101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:15.313271 (+ 170us) server_negotiation.cc:407] Connection header received
0926 06:57:15.313789 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:15.313790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:15.313803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:15.313860 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:15.314973 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.315563 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.316551 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.316760 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.316842 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:15.318914 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:15.318915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:15.318916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:15.318924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:15.320366 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.320785 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.320786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.320787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.320844 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.321082 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.321083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.321083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.321209 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:15.321279 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:15.321575 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 06:57:15.321654 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:57:15.321772 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 1703
I20260926 06:57:15.321862 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:15.321945 1834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:15.312988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:34595 (local address 127.1.74.67:37419)
0926 06:57:15.313158 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:15.313194 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:15.313689 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:15.313938 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:15.313941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:15.314632 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:15.314878 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:15.314883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.315653 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.315654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:15.316453 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:15.316455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.316851 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.316852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:15.316968 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:15.317454 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:15.317462 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:15.318814 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:15.320462 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:15.320468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:15.320471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:15.320696 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:15.320906 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:15.320907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:15.320908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:15.321017 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:15.321271 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:15.321276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:15.321414 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 06:57:15.321549 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 06:57:15.321705 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260926 06:57:15.322018 1321 external_mini_cluster.cc:1428] Setting key daacc9b9669429a4cd7b6d6ef918b3a3
I20260926 06:57:15.322614 1832 heartbeater.cc:347] Connected to a master server at 127.1.74.126:34595
I20260926 06:57:15.322731 1832 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:15.322933 1832 heartbeater.cc:511] Master 127.1.74.126:34595 requested a full tablet report, sending...
I20260926 06:57:15.323323 1372 ts_manager.cc:194] Registered new tserver with Master: 41fe8b4e71bb4bdc942983ae068401c7 (127.1.74.67:36923)
I20260926 06:57:15.323668 1372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:37419
I20260926 06:57:15.334146 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:15.356257 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:15.348840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.1.74.65:35217)
0926 06:57:15.349002 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:15.349008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:15.349021 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:57:15.349051 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:15.349054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:15.349075 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:15.349158 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:15.349704 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.350426 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.351159 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.351542 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.351636 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:15.353832 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:15.353838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:15.353839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:15.353845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:15.355217 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.355614 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.355618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.355618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.355674 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.355826 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.355827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.355827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.355952 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:15.356076 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:15.356116 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 06:57:15.356178 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
W20260926 06:57:15.356688 1478 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38120
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:15.379114 1839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:15.372212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.1.74.65:35217)
0926 06:57:15.372278 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:15.372280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:15.372363 (+ 83us) server_negotiation.cc:407] Connection header received
0926 06:57:15.372441 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:15.372441 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:15.372460 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:15.372523 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:15.373095 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.373697 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.374433 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:15.374643 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:15.374715 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:15.376902 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:15.376903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:15.376903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:15.376910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:15.378176 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.378468 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.378468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.378469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.378514 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:15.378720 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:15.378721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:15.378721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:15.378815 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:15.378897 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:15.378989 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 06:57:15.379048 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:57:15.379691 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 1435
I20260926 06:57:15.385911 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 1569
I20260926 06:57:15.390748 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 1703
I20260926 06:57:15.394920 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 1341
2026-09-26T06:57:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1895 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1846](info): set up 2 sockets
Sep 26 06:57:15 dist-test-slave-sfvt krb5kdc[1846](info): commencing operation
krb5kdc: starting...
W20260926 06:57:16.426875 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:16 dist-test-slave-sfvt krb5kdc[1846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:16.542604 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36851
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:36851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:16.622397 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:16.622638 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:16.622676 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:16.623884 1859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:16.623947 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:16.623970 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:16.623991 1859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:16.624018 1859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:16.625447 1859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:36851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:16.625721 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:16.625932 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:16.630579 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:16.630586 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:16.630795 1859 server_base.cc:1034] running on GCE node
W20260926 06:57:16.630609 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:16.631459 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:16.631819 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:16.632972 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790405836632964 us; error 38 us; skew 500 ppm
Sep 26 06:57:16 dist-test-slave-sfvt krb5kdc[1846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:16.635371 1859 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:16.635829 1859 webserver.cc:460] Webserver started at http://127.1.74.126:42805/ using document root <none> and password file <none>
I20260926 06:57:16.636051 1859 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:16.636113 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:16.636245 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:16.637244 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "1ee01ad371ce42f08fd1a86451c46bca"
format_stamp: "Formatted at 2026-09-26 06:57:16 on dist-test-slave-sfvt"
server_key: "9bc3e1d484f3c463bd2e62f77120a79c"
server_key_iv: "8ed20699d7d07a24824818dd91bc8355"
server_key_version: "encryptionkey@0"
I20260926 06:57:16.637595 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "1ee01ad371ce42f08fd1a86451c46bca"
format_stamp: "Formatted at 2026-09-26 06:57:16 on dist-test-slave-sfvt"
server_key: "9bc3e1d484f3c463bd2e62f77120a79c"
server_key_iv: "8ed20699d7d07a24824818dd91bc8355"
server_key_version: "encryptionkey@0"
I20260926 06:57:16.638844 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:16.639572 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:16.639729 1859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:16.639818 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "1ee01ad371ce42f08fd1a86451c46bca"
format_stamp: "Formatted at 2026-09-26 06:57:16 on dist-test-slave-sfvt"
server_key: "9bc3e1d484f3c463bd2e62f77120a79c"
server_key_iv: "8ed20699d7d07a24824818dd91bc8355"
server_key_version: "encryptionkey@0"
I20260926 06:57:16.639909 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:16.668990 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:16.670868 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:16.671029 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:16.674926 1859 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:36851
I20260926 06:57:16.674964 1926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:36851 every 8 connection(s)
I20260926 06:57:16.675334 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:16.676175 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:16.677736 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 1859
I20260926 06:57:16.677802 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:16.677959 1321 external_mini_cluster.cc:1428] Setting key b1e9cbfeaed9ee49970448dd5b0a8db6
I20260926 06:57:16.678776 1927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca: Bootstrap starting.
I20260926 06:57:16.679497 1927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:16.679796 1927 log.cc:824] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:16.680579 1927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca: No bootstrap required, opened a new log
I20260926 06:57:16.682793 1927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee01ad371ce42f08fd1a86451c46bca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36851 } }
I20260926 06:57:16.682955 1927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:16.682996 1927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee01ad371ce42f08fd1a86451c46bca, State: Initialized, Role: FOLLOWER
I20260926 06:57:16.683151 1927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee01ad371ce42f08fd1a86451c46bca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36851 } }
I20260926 06:57:16.683254 1927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:16.683329 1927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:16.683377 1927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:16.683965 1927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee01ad371ce42f08fd1a86451c46bca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36851 } }
I20260926 06:57:16.684080 1927 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee01ad371ce42f08fd1a86451c46bca; no voters:
I20260926 06:57:16.684248 1927 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:16.684281 1932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:16.684377 1932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [term 1 LEADER]: Becoming Leader. State: Replica: 1ee01ad371ce42f08fd1a86451c46bca, State: Running, Role: LEADER
I20260926 06:57:16.684455 1932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee01ad371ce42f08fd1a86451c46bca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36851 } }
I20260926 06:57:16.684494 1927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:16.684738 1932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee01ad371ce42f08fd1a86451c46bca. Latest consensus state: current_term: 1 leader_uuid: "1ee01ad371ce42f08fd1a86451c46bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee01ad371ce42f08fd1a86451c46bca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36851 } } }
I20260926 06:57:16.684816 1932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:16.684752 1933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ee01ad371ce42f08fd1a86451c46bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee01ad371ce42f08fd1a86451c46bca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36851 } } }
I20260926 06:57:16.684965 1933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca [sys.catalog]: This master's current role is: LEADER
Sep 26 06:57:16 dist-test-slave-sfvt krb5kdc[1846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
W20260926 06:57:16.686203 1947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:16.686254 1947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:16.686303 1937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:16.686455 1937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:16.687942 1937 catalog_manager.cc:1347] Generated new cluster ID: 72562720cf024d06aa907aca937ac009
I20260926 06:57:16.687992 1937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:16.689409 1930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:16.678479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.1.74.126:36851)
0926 06:57:16.678657 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:16.678666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:16.678696 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:57:16.679277 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:16.679287 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:16.679346 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:16.679507 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:16.680193 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:16.681586 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:16.682316 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:16.682950 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:16.683074 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:16.685735 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:16.685739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:16.685739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:16.685750 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:16.687823 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:16.688223 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:16.688226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:16.688226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:16.688323 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:16.688533 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:16.688534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:16.688534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:16.688807 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:16.688941 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:16.689085 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 06:57:16.689203 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260926 06:57:16.700280 1937 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:16.701040 1937 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:16.705693 1937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ee01ad371ce42f08fd1a86451c46bca: Generated new TSK 0
I20260926 06:57:16.705893 1937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:16.730201 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36851
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:16.809500 1953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:16.809716 1953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:16.809741 1953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:16.810900 1953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:16.810940 1953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:16.810995 1953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:16.812808 1953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36851
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:16.813073 1953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:16.813318 1953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:16.818666 1958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:16.818630 1959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:16.818630 1961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:16.819039 1953 server_base.cc:1034] running on GCE node
I20260926 06:57:16.819334 1953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:16.819715 1953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:16.820863 1953 hybrid_clock.cc:648] HybridClock initialized: now 1790405836820851 us; error 31 us; skew 500 ppm
Sep 26 06:57:16 dist-test-slave-sfvt krb5kdc[1846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:16.823439 1953 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:16.823871 1953 webserver.cc:460] Webserver started at http://127.1.74.65:42847/ using document root <none> and password file <none>
I20260926 06:57:16.824187 1953 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:16.824270 1953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:16.824434 1953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:16.825492 1953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "b30f39e863ff42afb750294763dea438"
format_stamp: "Formatted at 2026-09-26 06:57:16 on dist-test-slave-sfvt"
server_key: "ddbd9d088443fb62631bcd7e98951070"
server_key_iv: "4b6f88b4cb9f753ea94622c44c9e97d3"
server_key_version: "encryptionkey@0"
I20260926 06:57:16.825855 1953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "b30f39e863ff42afb750294763dea438"
format_stamp: "Formatted at 2026-09-26 06:57:16 on dist-test-slave-sfvt"
server_key: "ddbd9d088443fb62631bcd7e98951070"
server_key_iv: "4b6f88b4cb9f753ea94622c44c9e97d3"
server_key_version: "encryptionkey@0"
I20260926 06:57:16.827214 1953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:16.828055 1968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:16.828254 1953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:16.828351 1953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "b30f39e863ff42afb750294763dea438"
format_stamp: "Formatted at 2026-09-26 06:57:16 on dist-test-slave-sfvt"
server_key: "ddbd9d088443fb62631bcd7e98951070"
server_key_iv: "4b6f88b4cb9f753ea94622c44c9e97d3"
server_key_version: "encryptionkey@0"
I20260926 06:57:16.828438 1953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:16.837896 1953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:16.840178 1953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:16.840332 1953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:16.840541 1953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:16.840852 1953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:16.840902 1953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:16.840942 1953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:16.840969 1953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:16.847563 1953 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:33657
I20260926 06:57:16.847615 2081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:33657 every 8 connection(s)
I20260926 06:57:16.847939 1953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:57:16 dist-test-slave-sfvt krb5kdc[1846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:16.854581 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 1953
I20260926 06:57:16.854681 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:16.854853 1321 external_mini_cluster.cc:1428] Setting key f797b722ae69d1484931e754b2bf3a5a
I20260926 06:57:16.857941 1930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:16.848683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:45279 (local address 127.1.74.126:36851)
0926 06:57:16.848730 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:16.848733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:16.848957 (+ 224us) server_negotiation.cc:407] Connection header received
0926 06:57:16.849861 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:16.849863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:16.849874 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:16.849940 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:16.850581 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:16.851100 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:16.852144 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:16.852350 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:16.852413 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:16.854452 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:16.854453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:16.854453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:16.854461 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:16.856356 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:16.856833 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:16.856834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:16.856835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:16.856901 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:16.857232 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:16.857233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:16.857233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:16.857358 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:16.857414 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:16.857815 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 06:57:16.857879 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:57:16.858359 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:16.848626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36851 (local address 127.1.74.65:45279)
0926 06:57:16.848819 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:16.848861 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:16.849742 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:16.850015 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:16.850019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:16.850262 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:16.850446 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:16.850450 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:16.851188 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:16.851189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:16.852038 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:16.852040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:16.852428 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:16.852429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:16.852529 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:16.852952 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:16.852959 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:16.854346 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:16.856451 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:16.856457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:16.856458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:16.856737 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:16.856977 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:16.856978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:16.856978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:16.857147 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:16.857427 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:16.857433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:16.857654 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 06:57:16.857777 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 06:57:16.857944 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260926 06:57:16.858930 2082 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36851
I20260926 06:57:16.859054 2082 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:16.859287 2082 heartbeater.cc:511] Master 127.1.74.126:36851 requested a full tablet report, sending...
I20260926 06:57:16.859982 1891 ts_manager.cc:194] Registered new tserver with Master: b30f39e863ff42afb750294763dea438 (127.1.74.65:33657)
I20260926 06:57:16.860980 1891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:45279
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:16.881779 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36851
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:16.966187 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:16.966408 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:16.966434 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:16.967607 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:16.967650 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:16.967679 2087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:16.969081 2087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36851
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:16.969269 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:16.969442 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:16.974435 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:16.974460 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:16.974434 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:16.974817 2087 server_base.cc:1034] running on GCE node
I20260926 06:57:16.975026 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:16.975212 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:16.976394 2087 hybrid_clock.cc:648] HybridClock initialized: now 1790405836976382 us; error 27 us; skew 500 ppm
Sep 26 06:57:16 dist-test-slave-sfvt krb5kdc[1846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:16.978874 2087 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:16.979328 2087 webserver.cc:460] Webserver started at http://127.1.74.66:35805/ using document root <none> and password file <none>
I20260926 06:57:16.979552 2087 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:16.979621 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:16.979768 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:16.980795 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "991f1fb528424251812332062cb3357b"
format_stamp: "Formatted at 2026-09-26 06:57:16 on dist-test-slave-sfvt"
server_key: "cf61471db963ffaba71262d2cbc571ea"
server_key_iv: "ef4c61aca3bdff96e4a29bc0ebc3bafb"
server_key_version: "encryptionkey@0"
I20260926 06:57:16.981153 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "991f1fb528424251812332062cb3357b"
format_stamp: "Formatted at 2026-09-26 06:57:16 on dist-test-slave-sfvt"
server_key: "cf61471db963ffaba71262d2cbc571ea"
server_key_iv: "ef4c61aca3bdff96e4a29bc0ebc3bafb"
server_key_version: "encryptionkey@0"
I20260926 06:57:16.982445 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:16.983173 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:16.983358 2087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:16.983453 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "991f1fb528424251812332062cb3357b"
format_stamp: "Formatted at 2026-09-26 06:57:16 on dist-test-slave-sfvt"
server_key: "cf61471db963ffaba71262d2cbc571ea"
server_key_iv: "ef4c61aca3bdff96e4a29bc0ebc3bafb"
server_key_version: "encryptionkey@0"
I20260926 06:57:16.983533 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:16.993072 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:16.994853 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:16.994995 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:16.995195 2087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:16.995512 2087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:16.995563 2087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:16.995594 2087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:16.995630 2087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:17.001223 2087 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:37355
I20260926 06:57:17.001327 2215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:37355 every 8 connection(s)
I20260926 06:57:17.001586 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:17.006160 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 2087
I20260926 06:57:17.006280 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:17.006469 1321 external_mini_cluster.cc:1428] Setting key e54b6d379349d5818d3848f8e1ef5bc0
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[1846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405836, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:17.011032 1930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.002287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:36713 (local address 127.1.74.126:36851)
0926 06:57:17.002327 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.002329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.002507 (+ 178us) server_negotiation.cc:407] Connection header received
0926 06:57:17.003503 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.003504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.003515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.003567 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:17.004167 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.004670 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.005635 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.005815 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.005873 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.008209 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:17.008210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:17.008211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:17.008219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:17.009697 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.010080 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.010081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.010082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.010141 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.010403 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.010404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.010405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.010520 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:17.010594 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.010863 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.010943 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:57:17.011226 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.002244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36851 (local address 127.1.74.66:36713)
0926 06:57:17.002396 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:17.002437 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:17.003400 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:17.003629 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:17.003633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:17.003882 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:17.004059 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.004063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.004752 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.004752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.005535 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.005537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.005890 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.005890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.005987 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.006557 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:17.006567 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:17.008122 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:17.009772 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:17.009777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:17.009785 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:17.009990 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:17.010221 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:17.010222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:17.010222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:17.010325 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:17.010585 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:17.010590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:17.010716 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 06:57:17.010836 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 06:57:17.010987 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260926 06:57:17.011745 2216 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36851
I20260926 06:57:17.011854 2216 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:17.012039 2216 heartbeater.cc:511] Master 127.1.74.126:36851 requested a full tablet report, sending...
I20260926 06:57:17.012400 1891 ts_manager.cc:194] Registered new tserver with Master: 991f1fb528424251812332062cb3357b (127.1.74.66:37355)
I20260926 06:57:17.012959 1891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:36713
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:17.034025 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36851
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:17.113744 2221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:17.113973 2221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:17.114012 2221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:17.115182 2221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:17.115249 2221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:17.115324 2221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:17.116784 2221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36851
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:17.117029 2221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:17.117254 2221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:17.122231 2227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:17.122359 2221 server_base.cc:1034] running on GCE node
W20260926 06:57:17.122249 2229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:17.122471 2226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:17.122833 2221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:17.123106 2221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:17.124292 2221 hybrid_clock.cc:648] HybridClock initialized: now 1790405837124268 us; error 38 us; skew 500 ppm
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[1846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405837, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:17.126734 2221 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:17.127137 2221 webserver.cc:460] Webserver started at http://127.1.74.67:43981/ using document root <none> and password file <none>
I20260926 06:57:17.127389 2221 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:17.127454 2221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:17.127597 2221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:17.128664 2221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac"
format_stamp: "Formatted at 2026-09-26 06:57:17 on dist-test-slave-sfvt"
server_key: "e0f8ccd554c8cf7c15ca3de7b4e14431"
server_key_iv: "a84cbf6ee2d852bb27400ef527939b26"
server_key_version: "encryptionkey@0"
I20260926 06:57:17.129046 2221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac"
format_stamp: "Formatted at 2026-09-26 06:57:17 on dist-test-slave-sfvt"
server_key: "e0f8ccd554c8cf7c15ca3de7b4e14431"
server_key_iv: "a84cbf6ee2d852bb27400ef527939b26"
server_key_version: "encryptionkey@0"
I20260926 06:57:17.130484 2221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:17.131337 2236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:17.131513 2221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:17.131603 2221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac"
format_stamp: "Formatted at 2026-09-26 06:57:17 on dist-test-slave-sfvt"
server_key: "e0f8ccd554c8cf7c15ca3de7b4e14431"
server_key_iv: "a84cbf6ee2d852bb27400ef527939b26"
server_key_version: "encryptionkey@0"
I20260926 06:57:17.131690 2221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:17.148873 2221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:17.150655 2221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:17.150815 2221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:17.151037 2221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:17.151427 2221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:17.151480 2221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:17.151521 2221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:17.151549 2221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:17.157241 2221 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:41389
I20260926 06:57:17.157279 2349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:41389 every 8 connection(s)
I20260926 06:57:17.157609 2221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:57:17.158406 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 2221
I20260926 06:57:17.158506 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:17.158687 1321 external_mini_cluster.cc:1428] Setting key cad2e6ff7ee2e5563fe017cd9ecb6e1b
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[1846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405837, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:17.166886 1930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.158304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:51319 (local address 127.1.74.126:36851)
0926 06:57:17.158565 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.158567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.158646 (+ 79us) server_negotiation.cc:407] Connection header received
0926 06:57:17.159886 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.159887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.159898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.159967 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:17.160546 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.161076 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.162094 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.162288 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.162354 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.164515 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:17.164516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:17.164516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:17.164523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:17.165759 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.166095 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.166096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.166096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.166145 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.166356 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.166357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.166357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.166461 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:17.166526 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.166770 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.166824 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260926 06:57:17.167088 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.158283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36851 (local address 127.1.74.67:51319)
0926 06:57:17.158517 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:17.158565 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:17.159767 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:17.160024 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:17.160027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:17.160282 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:17.160461 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.160466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.161157 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.161158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.161996 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.161997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.162374 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.162375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.162471 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.162927 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:17.162935 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:17.164402 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:17.165830 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:17.165835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:17.165836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:17.166033 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:17.166212 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:17.166213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:17.166213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:17.166303 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:17.166530 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:17.166534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:17.166637 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 06:57:17.166742 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 06:57:17.166851 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
I20260926 06:57:17.167554 2350 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36851
I20260926 06:57:17.167642 2350 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:17.167768 2350 heartbeater.cc:511] Master 127.1.74.126:36851 requested a full tablet report, sending...
I20260926 06:57:17.168087 1891 ts_manager.cc:194] Registered new tserver with Master: dc7e8efd00a543b591bc1ff6ac7c1aac (127.1.74.67:41389)
I20260926 06:57:17.168409 1891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:51319
I20260926 06:57:17.170498 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:57:17.177500 1930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.172018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.1.74.126:36851)
0926 06:57:17.172078 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.172080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.172163 (+ 83us) server_negotiation.cc:407] Connection header received
0926 06:57:17.172220 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.172221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.172229 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.172275 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:17.172777 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.173224 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.173878 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.174061 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.174108 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.175145 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:17.175146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:17.175146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:17.175151 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:17.176521 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.176805 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.176805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.176806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.176851 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.177004 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.177004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.177005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.177114 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:17.177182 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.177295 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.177367 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:57:17.179751 1891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:17.181242 1891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:17.187813 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.183302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:33657 (local address 127.0.0.1:60106)
0926 06:57:17.183422 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:17.183459 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:17.183535 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:17.183964 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:17.183967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:17.183968 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:17.184065 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.184067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.185456 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.185457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.186545 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.186547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.187533 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.187533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.187632 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.187646 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:57:17.187721 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:57:17.187759 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":102,"thread_start_us":56,"threads_started":1}
I20260926 06:57:17.187891 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.183405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.1.74.65:33657)
0926 06:57:17.183660 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.183665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.183676 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:57:17.183718 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.183720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.183737 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.184141 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:17.184218 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.185334 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.186639 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.187392 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.187532 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.187597 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.187746 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.187784 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260926 06:57:17.188282 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.183615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:41389 (local address 127.0.0.1:41544)
0926 06:57:17.183825 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:17.183833 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:17.183872 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:17.184068 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:17.184068 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:17.184070 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:17.184127 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.184128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.185428 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.185429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.186778 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.186779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.188046 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.188047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.188125 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.188128 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:17.188170 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 06:57:17.188227 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":61,"thread_start_us":47,"threads_started":1}
I20260926 06:57:17.188565 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.183779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.1.74.67:41389)
0926 06:57:17.183937 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.183941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.183950 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:17.183981 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.183984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.183997 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.184072 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:17.184181 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.185356 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.186860 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.187570 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.187630 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.187675 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.188453 (+ 778us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.188495 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260926 06:57:17.188563 2016 tablet_service.cc:1467] Processing CreateTablet for tablet 865bfeb87cba4b30b8fcdf3021f22144 (DEFAULT_TABLE table=test-table [id=a2270cc8b751448eb2b85f8336c2353e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:17.188755 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.183506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:37355 (local address 127.0.0.1:59562)
0926 06:57:17.183693 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:17.183702 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:17.183740 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:17.183967 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:17.183968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:17.183971 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:17.184090 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.184106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.185384 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.185385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.186772 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.186774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.188541 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.188542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.188605 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.188609 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:17.188672 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:57:17.188714 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":59,"thread_start_us":30,"threads_started":1}
I20260926 06:57:17.188796 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.183623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.1.74.66:37355)
0926 06:57:17.183797 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.183802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.183812 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:17.183852 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.183855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.183870 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.183947 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:17.184209 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.185284 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.187724 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.188432 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.188535 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.188589 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.188686 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.188727 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260926 06:57:17.188930 2016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865bfeb87cba4b30b8fcdf3021f22144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:17.189400 2150 tablet_service.cc:1467] Processing CreateTablet for tablet 865bfeb87cba4b30b8fcdf3021f22144 (DEFAULT_TABLE table=test-table [id=a2270cc8b751448eb2b85f8336c2353e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:17.189433 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 865bfeb87cba4b30b8fcdf3021f22144 (DEFAULT_TABLE table=test-table [id=a2270cc8b751448eb2b85f8336c2353e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:17.189692 2150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865bfeb87cba4b30b8fcdf3021f22144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:17.189726 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865bfeb87cba4b30b8fcdf3021f22144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:17.192416 2365 tablet_bootstrap.cc:492] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438: Bootstrap starting.
I20260926 06:57:17.192896 2366 tablet_bootstrap.cc:492] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b: Bootstrap starting.
I20260926 06:57:17.193051 2365 tablet_bootstrap.cc:654] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:17.193748 2364 tablet_bootstrap.cc:492] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac: Bootstrap starting.
I20260926 06:57:17.193835 2365 log.cc:824] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:17.193920 2366 tablet_bootstrap.cc:654] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:17.194406 2364 tablet_bootstrap.cc:654] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:17.194487 2366 log.cc:824] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:17.194633 2365 tablet_bootstrap.cc:492] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438: No bootstrap required, opened a new log
I20260926 06:57:17.194734 2365 ts_tablet_manager.cc:1397] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:17.195006 2364 log.cc:824] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:17.195340 2366 tablet_bootstrap.cc:492] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b: No bootstrap required, opened a new log
I20260926 06:57:17.195394 2366 ts_tablet_manager.cc:1397] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:57:17.195653 2364 tablet_bootstrap.cc:492] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac: No bootstrap required, opened a new log
I20260926 06:57:17.195693 2364 ts_tablet_manager.cc:1397] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:17.197029 2364 raft_consensus.cc:348] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } }
I20260926 06:57:17.197155 2364 raft_consensus.cc:374] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:17.197177 2364 raft_consensus.cc:729] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7e8efd00a543b591bc1ff6ac7c1aac, State: Initialized, Role: FOLLOWER
I20260926 06:57:17.197148 2365 raft_consensus.cc:348] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } }
I20260926 06:57:17.197306 2365 raft_consensus.cc:374] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:17.197294 2364 consensus_queue.cc:260] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } }
I20260926 06:57:17.197340 2365 raft_consensus.cc:729] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30f39e863ff42afb750294763dea438, State: Initialized, Role: FOLLOWER
I20260926 06:57:17.197450 2365 consensus_queue.cc:260] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } }
I20260926 06:57:17.197568 2364 ts_tablet_manager.cc:1428] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:17.197685 2350 heartbeater.cc:503] Master 127.1.74.126:36851 was elected leader, sending a full tablet report...
I20260926 06:57:17.197714 2365 ts_tablet_manager.cc:1428] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:57:17.197659 2366 raft_consensus.cc:348] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } }
I20260926 06:57:17.197818 2082 heartbeater.cc:503] Master 127.1.74.126:36851 was elected leader, sending a full tablet report...
I20260926 06:57:17.197813 2366 raft_consensus.cc:374] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:17.197849 2366 raft_consensus.cc:729] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991f1fb528424251812332062cb3357b, State: Initialized, Role: FOLLOWER
I20260926 06:57:17.198400 2366 consensus_queue.cc:260] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } }
I20260926 06:57:17.198673 2366 ts_tablet_manager.cc:1428] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:57:17.198756 2216 heartbeater.cc:503] Master 127.1.74.126:36851 was elected leader, sending a full tablet report...
W20260926 06:57:17.252134 2217 tablet.cc:2369] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:57:17.348670 2083 tablet.cc:2369] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:57:17.408185 2351 tablet.cc:2369] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:17.596695 2371 raft_consensus.cc:482] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:17.596869 2371 raft_consensus.cc:504] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } }
I20260926 06:57:17.597306 2371 leader_election.cc:288] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 991f1fb528424251812332062cb3357b (127.1.74.66:37355), dc7e8efd00a543b591bc1ff6ac7c1aac (127.1.74.67:41389)
I20260926 06:57:17.600464 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.597842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:34099 (local address 127.1.74.67:41389)
0926 06:57:17.597930 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.597932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.597941 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:17.597973 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.597974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.597983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.598048 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:17.598209 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.598712 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.599747 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.600119 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.600206 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.600217 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.600371 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.600403 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 06:57:17.600474 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.597730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:41389 (local address 127.1.74.65:34099)
0926 06:57:17.597842 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:17.597850 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:17.597891 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:17.598052 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:17.598052 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:17.598054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:17.598133 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.598139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.598809 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.598810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.599653 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.599655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.600206 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.600206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.600275 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.600279 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:17.600350 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:57:17.600389 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":16,"thread_start_us":58,"threads_started":1}
I20260926 06:57:17.600541 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.597601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:46265 (local address 127.1.74.66:37355)
0926 06:57:17.597656 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.597658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.597789 (+ 131us) server_negotiation.cc:407] Connection header received
0926 06:57:17.597829 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.597830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.597842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.597931 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:17.598093 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.598580 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.599821 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.600175 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.600253 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.600262 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.600428 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.600464 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:57:17.600651 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.597516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:37355 (local address 127.1.74.65:46265)
0926 06:57:17.597690 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:17.597702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:17.597775 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:17.597939 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:17.597940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:17.597943 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:17.598016 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.598018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.598717 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.598718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.599729 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:17.599731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.600265 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.600266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:17.600357 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.600360 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:17.600424 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:57:17.600464 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":19,"thread_start_us":98,"threads_started":1}
I20260926 06:57:17.600878 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865bfeb87cba4b30b8fcdf3021f22144" candidate_uuid: "b30f39e863ff42afb750294763dea438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" is_pre_election: true
I20260926 06:57:17.600957 2170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865bfeb87cba4b30b8fcdf3021f22144" candidate_uuid: "b30f39e863ff42afb750294763dea438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991f1fb528424251812332062cb3357b" is_pre_election: true
I20260926 06:57:17.601053 2304 raft_consensus.cc:2454] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b30f39e863ff42afb750294763dea438 in term 0.
I20260926 06:57:17.601099 2170 raft_consensus.cc:2454] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b30f39e863ff42afb750294763dea438 in term 0.
I20260926 06:57:17.601287 1969 leader_election.cc:302] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 991f1fb528424251812332062cb3357b, b30f39e863ff42afb750294763dea438; no voters:
I20260926 06:57:17.601450 2371 raft_consensus.cc:2781] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:17.601518 2371 raft_consensus.cc:482] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:17.601536 2371 raft_consensus.cc:3037] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:17.602253 2371 raft_consensus.cc:504] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } }
I20260926 06:57:17.602387 2371 leader_election.cc:288] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [CANDIDATE]: Term 1 election: Requested vote from peers 991f1fb528424251812332062cb3357b (127.1.74.66:37355), dc7e8efd00a543b591bc1ff6ac7c1aac (127.1.74.67:41389)
I20260926 06:57:17.602574 2170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865bfeb87cba4b30b8fcdf3021f22144" candidate_uuid: "b30f39e863ff42afb750294763dea438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "991f1fb528424251812332062cb3357b"
I20260926 06:57:17.602633 2304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865bfeb87cba4b30b8fcdf3021f22144" candidate_uuid: "b30f39e863ff42afb750294763dea438" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac"
I20260926 06:57:17.602648 2170 raft_consensus.cc:3037] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:17.602713 2304 raft_consensus.cc:3037] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:17.603461 2170 raft_consensus.cc:2454] T 865bfeb87cba4b30b8fcdf3021f22144 P 991f1fb528424251812332062cb3357b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30f39e863ff42afb750294763dea438 in term 1.
I20260926 06:57:17.603467 2304 raft_consensus.cc:2454] T 865bfeb87cba4b30b8fcdf3021f22144 P dc7e8efd00a543b591bc1ff6ac7c1aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30f39e863ff42afb750294763dea438 in term 1.
I20260926 06:57:17.603611 1969 leader_election.cc:302] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 991f1fb528424251812332062cb3357b, b30f39e863ff42afb750294763dea438; no voters:
I20260926 06:57:17.603694 2371 raft_consensus.cc:2781] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:17.603825 2371 raft_consensus.cc:686] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [term 1 LEADER]: Becoming Leader. State: Replica: b30f39e863ff42afb750294763dea438, State: Running, Role: LEADER
I20260926 06:57:17.603920 2371 consensus_queue.cc:237] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } }
I20260926 06:57:17.604786 1890 catalog_manager.cc:5495] T 865bfeb87cba4b30b8fcdf3021f22144 P b30f39e863ff42afb750294763dea438 reported cstate change: term changed from 0 to 1, leader changed from <none> to b30f39e863ff42afb750294763dea438 (127.1.74.65). New cstate: current_term: 1 leader_uuid: "b30f39e863ff42afb750294763dea438" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b30f39e863ff42afb750294763dea438" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "991f1fb528424251812332062cb3357b" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 37355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc7e8efd00a543b591bc1ff6ac7c1aac" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41389 } health_report { overall_health: UNKNOWN } } }
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[1846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:17.633488 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.626052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.1.74.65:33657)
0926 06:57:17.626135 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.626138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.626246 (+ 108us) server_negotiation.cc:407] Connection header received
0926 06:57:17.626316 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.626317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.626328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.626416 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:17.626990 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.627520 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.628297 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.628504 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.628577 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.630851 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:17.630864 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:17.630865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:17.630873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:17.632429 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.632753 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.632756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.632756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.632817 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.632975 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.632976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.632976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.633105 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:17.633204 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.633290 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.633350 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[1846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[1846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:17.655475 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.648815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.1.74.65:33657)
0926 06:57:17.648895 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.648897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.648989 (+ 92us) server_negotiation.cc:407] Connection header received
0926 06:57:17.649031 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.649032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.649040 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.649088 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:17.649604 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.650068 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.650757 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.650935 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.650983 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.653204 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:17.653206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:17.653206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:17.653212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:17.654370 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.654689 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.654690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.654690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.654736 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.654888 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.654889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.654889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.654982 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:17.655070 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.655248 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.655377 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[1846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[1846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:17.677016 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:17.670107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.1.74.65:33657)
0926 06:57:17.670167 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:17.670170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:17.670255 (+ 85us) server_negotiation.cc:407] Connection header received
0926 06:57:17.670304 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:17.670305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:17.670314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:17.670364 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:17.670947 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.671452 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.672246 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:17.672429 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:17.672479 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:17.674527 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:17.674528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:17.674529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:17.674534 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:17.675894 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.676280 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.676281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.676281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.676332 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:17.676523 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:17.676524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:17.676524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:17.676606 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:17.676731 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:17.676831 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 06:57:17.676908 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 06:57:17.677435 1996 tablet_service.cc:2908] Rejecting scan request for tablet 865bfeb87cba4b30b8fcdf3021f22144: Uninitialized: clean time has not yet been initialized
I20260926 06:57:17.678613 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 1953
I20260926 06:57:17.683897 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 2087
I20260926 06:57:17.690266 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 2221
I20260926 06:57:17.695369 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 1859
2026-09-26T06:57:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2302 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[2389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[2389](info): set up 2 sockets
Sep 26 06:57:17 dist-test-slave-sfvt krb5kdc[2389](info): commencing operation
krb5kdc: starting...
W20260926 06:57:18.734964 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:18 dist-test-slave-sfvt krb5kdc[2389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:18.853312 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33881
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:33881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:18.935364 2402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:18.935631 2402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:18.935668 2402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:18.936808 2402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:18.936882 2402 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 06:57:18.936902 2402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:18.936923 2402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:18.936939 2402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:18.936959 2402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:18.938447 2402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:33881
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:18.938755 2402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:18.938979 2402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:18.943806 2410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:18.943813 2408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:18.943897 2407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:18.944048 2402 server_base.cc:1034] running on GCE node
I20260926 06:57:18.944294 2402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:18.944595 2402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:18.945765 2402 hybrid_clock.cc:648] HybridClock initialized: now 1790405838945761 us; error 40 us; skew 500 ppm
Sep 26 06:57:18 dist-test-slave-sfvt krb5kdc[2389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:18.948294 2402 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:18.948746 2402 webserver.cc:460] Webserver started at http://127.1.74.126:33699/ using document root <none> and password file <none>
I20260926 06:57:18.948930 2402 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:18.948968 2402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:18.949054 2402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:18.950033 2402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "b8d0af46cf334dfda545c2c18cbe1b05"
format_stamp: "Formatted at 2026-09-26 06:57:18 on dist-test-slave-sfvt"
server_key: "4d95059a8a293375116e76acebbd64af"
server_key_iv: "41b0e7bfbfd5be6d6b5c18059a2d8a0e"
server_key_version: "encryptionkey@0"
I20260926 06:57:18.950321 2402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "b8d0af46cf334dfda545c2c18cbe1b05"
format_stamp: "Formatted at 2026-09-26 06:57:18 on dist-test-slave-sfvt"
server_key: "4d95059a8a293375116e76acebbd64af"
server_key_iv: "41b0e7bfbfd5be6d6b5c18059a2d8a0e"
server_key_version: "encryptionkey@0"
I20260926 06:57:18.951601 2402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:18.952270 2417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:18.952447 2402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:18.952534 2402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "b8d0af46cf334dfda545c2c18cbe1b05"
format_stamp: "Formatted at 2026-09-26 06:57:18 on dist-test-slave-sfvt"
server_key: "4d95059a8a293375116e76acebbd64af"
server_key_iv: "41b0e7bfbfd5be6d6b5c18059a2d8a0e"
server_key_version: "encryptionkey@0"
I20260926 06:57:18.952589 2402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:18.969743 2402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:18.971529 2402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:18.971634 2402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:18.975961 2402 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:33881
I20260926 06:57:18.976044 2479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:33881 every 8 connection(s)
I20260926 06:57:18.976356 2402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:18.977033 2480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:18.978057 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 2402
I20260926 06:57:18.978132 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:18.978300 1321 external_mini_cluster.cc:1428] Setting key 67bf2fb0a003195f3b445c86c1974e85
I20260926 06:57:18.979651 2480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05: Bootstrap starting.
I20260926 06:57:18.980402 2480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:18.980695 2480 log.cc:824] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:18.981348 2480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05: No bootstrap required, opened a new log
I20260926 06:57:18.983700 2480 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d0af46cf334dfda545c2c18cbe1b05" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33881 } }
I20260926 06:57:18.983875 2480 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:18.983923 2480 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d0af46cf334dfda545c2c18cbe1b05, State: Initialized, Role: FOLLOWER
I20260926 06:57:18.984062 2480 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d0af46cf334dfda545c2c18cbe1b05" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33881 } }
I20260926 06:57:18.984155 2480 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:18.984193 2480 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:18.984247 2480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:18.984915 2480 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d0af46cf334dfda545c2c18cbe1b05" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33881 } }
I20260926 06:57:18.984998 2480 leader_election.cc:302] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8d0af46cf334dfda545c2c18cbe1b05; no voters:
I20260926 06:57:18.985194 2480 leader_election.cc:288] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:18.985219 2485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [term 1 FOLLOWER]: Leader election won for term 1
Sep 26 06:57:18 dist-test-slave-sfvt krb5kdc[2389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:18.985330 2485 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [term 1 LEADER]: Becoming Leader. State: Replica: b8d0af46cf334dfda545c2c18cbe1b05, State: Running, Role: LEADER
I20260926 06:57:18.985416 2485 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d0af46cf334dfda545c2c18cbe1b05" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33881 } }
I20260926 06:57:18.985445 2480 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:18.987411 2486 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8d0af46cf334dfda545c2c18cbe1b05. Latest consensus state: current_term: 1 leader_uuid: "b8d0af46cf334dfda545c2c18cbe1b05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d0af46cf334dfda545c2c18cbe1b05" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33881 } } }
I20260926 06:57:18.987532 2487 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8d0af46cf334dfda545c2c18cbe1b05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d0af46cf334dfda545c2c18cbe1b05" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33881 } } }
I20260926 06:57:18.988157 2486 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:18.988166 2487 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05 [sys.catalog]: This master's current role is: LEADER
W20260926 06:57:18.987979 2499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:18.988391 2499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:18.988463 2502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:18.988642 2502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:18.989598 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:18.978780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.1.74.126:33881)
0926 06:57:18.978946 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:18.978953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:18.979070 (+ 117us) server_negotiation.cc:407] Connection header received
0926 06:57:18.979555 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:18.979562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:18.979615 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:18.979729 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:18.980585 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:18.981540 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:18.982638 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:18.983029 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:18.983105 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:18.985817 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:18.985820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:18.985821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:18.985830 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:18.987815 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:18.988267 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:18.988270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:18.988270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:18.988378 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:18.988603 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:18.988604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:18.988605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:18.988905 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:18.989054 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:18.989179 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 06:57:18.989356 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260926 06:57:18.991659 2502 catalog_manager.cc:1347] Generated new cluster ID: 5fa96bb86bf541f8a3323c1102d35c7f
I20260926 06:57:18.991716 2502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:18.996393 2502 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:18.996970 2502 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:19.004416 2502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05: Generated new TSK 0
I20260926 06:57:19.004729 2502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:19.038961 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33881
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:19.134501 2506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:19.134733 2506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:19.134776 2506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:19.136034 2506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:19.136096 2506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:19.136132 2506 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:57:19.136190 2506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:19.137619 2506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.74.126:33881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:19.137877 2506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:19.138113 2506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:19.143170 2512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:19.143221 2511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:19.143306 2506 server_base.cc:1034] running on GCE node
W20260926 06:57:19.143404 2514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:19.143743 2506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:19.143993 2506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:19.145154 2506 hybrid_clock.cc:648] HybridClock initialized: now 1790405839145149 us; error 38 us; skew 500 ppm
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:19.147513 2506 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:19.147992 2506 webserver.cc:460] Webserver started at http://127.1.74.65:43529/ using document root <none> and password file <none>
I20260926 06:57:19.148218 2506 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:19.148283 2506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:19.148437 2506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:19.149473 2506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "f7fcc9a8cdfe491f971ddcbad355bdce"
format_stamp: "Formatted at 2026-09-26 06:57:19 on dist-test-slave-sfvt"
server_key: "1f5a24fd91d63273224c20267d24e9ac"
server_key_iv: "cf0d3a5824b78d8c1fbad591a122acb9"
server_key_version: "encryptionkey@0"
I20260926 06:57:19.149834 2506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "f7fcc9a8cdfe491f971ddcbad355bdce"
format_stamp: "Formatted at 2026-09-26 06:57:19 on dist-test-slave-sfvt"
server_key: "1f5a24fd91d63273224c20267d24e9ac"
server_key_iv: "cf0d3a5824b78d8c1fbad591a122acb9"
server_key_version: "encryptionkey@0"
I20260926 06:57:19.151165 2506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:19.151978 2521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.152168 2506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:19.152267 2506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "f7fcc9a8cdfe491f971ddcbad355bdce"
format_stamp: "Formatted at 2026-09-26 06:57:19 on dist-test-slave-sfvt"
server_key: "1f5a24fd91d63273224c20267d24e9ac"
server_key_iv: "cf0d3a5824b78d8c1fbad591a122acb9"
server_key_version: "encryptionkey@0"
I20260926 06:57:19.152350 2506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:19.184773 2506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:19.187117 2506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:19.187312 2506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:19.187816 2506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:19.187877 2506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.187933 2506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:19.187960 2506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:19.195205 2506 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:33965
I20260926 06:57:19.195355 2636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:33965 every 8 connection(s)
I20260926 06:57:19.195614 2506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:19.197444 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.188354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:57401 (local address 127.1.74.126:33881)
0926 06:57:19.188434 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.188437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.188599 (+ 162us) server_negotiation.cc:407] Connection header received
0926 06:57:19.189092 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.189094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.189109 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.189199 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:19.190083 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.190935 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.192115 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.192375 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.192467 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.194717 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:19.194719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:19.194719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:19.194727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:19.196333 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.196680 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.196681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.196681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.196731 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.196926 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.196927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.196927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.197024 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:19.197072 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.197283 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.197354 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:57:19.197580 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.188313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33881 (local address 127.1.74.65:57401)
0926 06:57:19.188464 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.188503 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.188996 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.189297 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.189302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.189695 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:19.189986 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.189992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.191033 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.191034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.192016 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.192018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.192487 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.192488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.192624 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.193132 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:19.193142 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:19.194621 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:19.196402 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.196406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.196407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.196614 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.196785 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.196785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.196785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.196874 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.197080 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:19.197084 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:19.197194 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.197274 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.197383 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260926 06:57:19.198088 2637 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33881
I20260926 06:57:19.198163 2637 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:19.198346 2637 heartbeater.cc:511] Master 127.1.74.126:33881 requested a full tablet report, sending...
I20260926 06:57:19.198897 2434 ts_manager.cc:194] Registered new tserver with Master: f7fcc9a8cdfe491f971ddcbad355bdce (127.1.74.65:33965)
I20260926 06:57:19.199748 2434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:57401
I20260926 06:57:19.204406 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.199300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:54305 (local address 127.1.74.126:33881)
0926 06:57:19.199340 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.199342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.199474 (+ 132us) server_negotiation.cc:407] Connection header received
0926 06:57:19.199498 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.199498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.199507 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.199577 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:19.199872 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.200347 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.201119 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.201293 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.201349 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.202130 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:19.202131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:19.202131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:19.202137 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:19.203565 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.203876 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.203877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.203877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.203923 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.204063 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.204063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.204063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.204147 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:19.204190 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.204290 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.204347 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:57:19.204514 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.199224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33881 (local address 127.1.74.65:54305)
0926 06:57:19.199399 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.199416 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.199465 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.199580 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.199581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.199729 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:19.199782 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.199783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.200465 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.200466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.201046 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.201048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.201365 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.201365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.201458 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.201799 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:19.201802 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:19.202057 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:19.203631 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.203632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.203632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.203785 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.203976 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.203976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.203977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.204024 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.204193 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:19.204193 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:19.204226 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.204275 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.204347 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260926 06:57:19.204540 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 2506
I20260926 06:57:19.204614 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:19.204756 1321 external_mini_cluster.cc:1428] Setting key 35700ed7bbfc185908660a0c570ec386
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:19.230875 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33881
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:19.312228 2646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:19.312494 2646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:19.312538 2646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:19.313699 2646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:19.313776 2646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:19.313822 2646 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:57:19.313864 2646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:19.315411 2646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.74.126:33881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:19.315680 2646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:19.315943 2646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:19.320910 2652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:19.320953 2654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:19.320911 2651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:19.321269 2646 server_base.cc:1034] running on GCE node
I20260926 06:57:19.321550 2646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:19.321810 2646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:19.323005 2646 hybrid_clock.cc:648] HybridClock initialized: now 1790405839322991 us; error 44 us; skew 500 ppm
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:19.325556 2646 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:19.326066 2646 webserver.cc:460] Webserver started at http://127.1.74.66:36533/ using document root <none> and password file <none>
I20260926 06:57:19.326287 2646 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:19.326352 2646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:19.326483 2646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:19.327595 2646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "7787eed3cf464f5c867f8e66bed92dd6"
format_stamp: "Formatted at 2026-09-26 06:57:19 on dist-test-slave-sfvt"
server_key: "f15fafb88de5705c189c76a94ec80a0d"
server_key_iv: "84abc6cf89cafd29b83ed625a2e52601"
server_key_version: "encryptionkey@0"
I20260926 06:57:19.327951 2646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "7787eed3cf464f5c867f8e66bed92dd6"
format_stamp: "Formatted at 2026-09-26 06:57:19 on dist-test-slave-sfvt"
server_key: "f15fafb88de5705c189c76a94ec80a0d"
server_key_iv: "84abc6cf89cafd29b83ed625a2e52601"
server_key_version: "encryptionkey@0"
I20260926 06:57:19.329217 2646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:57:19.329875 2661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.330047 2646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:19.330147 2646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "7787eed3cf464f5c867f8e66bed92dd6"
format_stamp: "Formatted at 2026-09-26 06:57:19 on dist-test-slave-sfvt"
server_key: "f15fafb88de5705c189c76a94ec80a0d"
server_key_iv: "84abc6cf89cafd29b83ed625a2e52601"
server_key_version: "encryptionkey@0"
I20260926 06:57:19.330224 2646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:19.339183 2646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:19.341722 2646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:19.341825 2646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:19.342406 2646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:19.342460 2646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.342495 2646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:19.342519 2646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:19.349641 2646 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:38171
I20260926 06:57:19.349973 2646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:19.350047 2776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:38171 every 8 connection(s)
I20260926 06:57:19.352123 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.342826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:52457 (local address 127.1.74.126:33881)
0926 06:57:19.342905 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.342908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.343142 (+ 234us) server_negotiation.cc:407] Connection header received
0926 06:57:19.343852 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.343853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.343864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.343944 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:19.344827 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.345313 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.346396 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.346600 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.346674 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.349185 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:19.349186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:19.349187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:19.349195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:19.350732 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.351161 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.351162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.351163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.351218 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.351492 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.351493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.351493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.351620 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:19.351696 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.351958 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.352034 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:57:19.352303 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.342805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33881 (local address 127.1.74.66:52457)
0926 06:57:19.342987 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.343038 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.343749 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.344027 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.344031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.344418 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:19.344710 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.344718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.345407 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.345408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.346305 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.346307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.346692 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.346693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.346819 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.347322 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:19.347330 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:19.349089 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:19.350829 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.350836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.350838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.351085 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.351322 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.351322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.351323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.351424 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.351687 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:19.351693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:19.351828 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.351926 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.352084 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260926 06:57:19.352793 2777 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33881
I20260926 06:57:19.352873 2777 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:19.353062 2777 heartbeater.cc:511] Master 127.1.74.126:33881 requested a full tablet report, sending...
I20260926 06:57:19.353415 2433 ts_manager.cc:194] Registered new tserver with Master: 7787eed3cf464f5c867f8e66bed92dd6 (127.1.74.66:38171)
I20260926 06:57:19.353871 2433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:52457
I20260926 06:57:19.355952 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 2646
I20260926 06:57:19.356061 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:19.356245 1321 external_mini_cluster.cc:1428] Setting key db758592a7cf5a7632b65c8364e22027
I20260926 06:57:19.359732 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.353855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:43887 (local address 127.1.74.126:33881)
0926 06:57:19.353905 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.353907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.353942 (+ 35us) server_negotiation.cc:407] Connection header received
0926 06:57:19.354036 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.354037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.354046 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.354090 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:19.354413 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.354975 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.355806 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.356092 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.356155 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.357168 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:19.357169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:19.357170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:19.357176 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:19.358527 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.358943 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.358944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.358945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.359005 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.359207 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.359207 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.359208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.359390 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:19.359445 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.359616 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.359677 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:57:19.359858 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.353729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33881 (local address 127.1.74.66:43887)
0926 06:57:19.353880 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.353892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.353954 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.354104 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.354105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.354275 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:19.354329 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.354331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.355074 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.355075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.355689 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.355691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.356209 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.356210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.356286 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.356711 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:19.356716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:19.357106 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:19.358638 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.358639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.358640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.358870 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.359080 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.359080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.359081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.359148 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.359455 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:19.359456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:19.359516 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.359590 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.359703 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:19.382113 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33881
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:19.462312 2786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:19.462508 2786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:19.462533 2786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:19.463827 2786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:19.463868 2786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:19.463889 2786 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:57:19.463929 2786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:19.465333 2786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.74.126:33881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:19.465577 2786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:19.465814 2786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:19.470696 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:19.470674 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:19.470683 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:19.471010 2786 server_base.cc:1034] running on GCE node
I20260926 06:57:19.471241 2786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:19.471484 2786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:19.472646 2786 hybrid_clock.cc:648] HybridClock initialized: now 1790405839472638 us; error 38 us; skew 500 ppm
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:19.480314 2786 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:19.480738 2786 webserver.cc:460] Webserver started at http://127.1.74.67:39859/ using document root <none> and password file <none>
I20260926 06:57:19.480953 2786 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:19.481022 2786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:19.481179 2786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:19.482209 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "adf3cb095ee5469fa75c149a9277de7a"
format_stamp: "Formatted at 2026-09-26 06:57:19 on dist-test-slave-sfvt"
server_key: "8b93879840e7f16bf20abefcfe885f62"
server_key_iv: "5b00c891b033ff6afb30dee728e07dbf"
server_key_version: "encryptionkey@0"
I20260926 06:57:19.482563 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "adf3cb095ee5469fa75c149a9277de7a"
format_stamp: "Formatted at 2026-09-26 06:57:19 on dist-test-slave-sfvt"
server_key: "8b93879840e7f16bf20abefcfe885f62"
server_key_iv: "5b00c891b033ff6afb30dee728e07dbf"
server_key_version: "encryptionkey@0"
I20260926 06:57:19.483918 2786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:19.484637 2801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.484807 2786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:19.484898 2786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "adf3cb095ee5469fa75c149a9277de7a"
format_stamp: "Formatted at 2026-09-26 06:57:19 on dist-test-slave-sfvt"
server_key: "8b93879840e7f16bf20abefcfe885f62"
server_key_iv: "5b00c891b033ff6afb30dee728e07dbf"
server_key_version: "encryptionkey@0"
I20260926 06:57:19.484979 2786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:19.505805 2786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:19.508175 2786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:19.508338 2786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:19.508822 2786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:19.508894 2786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.508940 2786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:19.508967 2786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.515594 2786 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:40561
I20260926 06:57:19.515641 2916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:40561 every 8 connection(s)
I20260926 06:57:19.515940 2786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405839, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:19.516611 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 2786
I20260926 06:57:19.516674 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:19.516840 1321 external_mini_cluster.cc:1428] Setting key a1b9adb26acddb41d82094d6d4a27548
I20260926 06:57:19.519111 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.509294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:38491 (local address 127.1.74.126:33881)
0926 06:57:19.509351 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.509353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.509553 (+ 200us) server_negotiation.cc:407] Connection header received
0926 06:57:19.510102 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.510103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.510116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.510181 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:19.511024 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.511866 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.513566 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.513818 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.513878 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.516528 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:19.516529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:19.516529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:19.516536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:19.518063 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.518400 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.518401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.518401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.518450 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.518632 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.518633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.518633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.518727 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:19.518776 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.518997 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.519051 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:57:19.519330 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.509295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33881 (local address 127.1.74.67:38491)
0926 06:57:19.509441 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.509476 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.509989 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.510252 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.510256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.510620 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:19.510915 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.510922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.511969 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.511971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.513444 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.513448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.513907 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.513908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.514071 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.514647 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:19.514659 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:19.516439 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:19.518129 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.518134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.518135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.518340 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.518491 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.518491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.518492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.518570 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.518781 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:19.518785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:19.518889 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.518967 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.519086 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260926 06:57:19.519786 2917 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33881
I20260926 06:57:19.519863 2917 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:19.520040 2917 heartbeater.cc:511] Master 127.1.74.126:33881 requested a full tablet report, sending...
I20260926 06:57:19.520358 2433 ts_manager.cc:194] Registered new tserver with Master: adf3cb095ee5469fa75c149a9277de7a (127.1.74.67:40561)
I20260926 06:57:19.520838 2433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:38491
I20260926 06:57:19.525502 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.520838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:49501 (local address 127.1.74.126:33881)
0926 06:57:19.520880 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.520882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.520950 (+ 68us) server_negotiation.cc:407] Connection header received
0926 06:57:19.521019 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.521020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.521028 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.521072 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:19.521403 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.521847 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.522469 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.522642 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.522689 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.523455 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:19.523456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:19.523456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:19.523462 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:19.524750 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.525012 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.525012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.525012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.525057 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.525179 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.525179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.525179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.525259 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:19.525303 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.525401 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.525449 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:57:19.525532 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.520758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33881 (local address 127.1.74.67:49501)
0926 06:57:19.520891 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.520902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.520963 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.521075 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.521076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.521267 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:19.521328 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.521330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.521930 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.521931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.522400 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.522402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.522700 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.522701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.522752 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.523068 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:19.523072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:19.523386 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:19.524813 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.524814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.524814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.524955 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.525107 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.525107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.525108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.525147 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.525307 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:19.525307 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:19.525339 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.525388 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.525457 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260926 06:57:19.528595 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:19.550436 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.543364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.1.74.126:33881)
0926 06:57:19.543422 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.543424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.543433 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:19.543454 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.543455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.543463 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.543529 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:19.544123 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.544614 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.545348 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.545537 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.545609 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.548045 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:19.548047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:19.548047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:19.548055 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:19.549328 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.549694 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.549695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.549696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.549749 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.549945 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.549946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.549946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.550055 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:19.550160 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.550255 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.550334 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:57:19.551939 2433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:19.553655 2433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:19.559760 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.556091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:33965 (local address 127.0.0.1:43504)
0926 06:57:19.556343 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.556385 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.556453 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.556724 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.556727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.556730 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:19.556793 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.556796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.557677 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.557678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.558611 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.558614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.559468 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.559469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.559548 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.559563 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.559639 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.559685 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":54,"threads_started":1}
I20260926 06:57:19.559810 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.556344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.1.74.65:33965)
0926 06:57:19.556467 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.556472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.556486 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:57:19.556548 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.556550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.556565 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.556643 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:19.556876 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.557591 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.558710 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.559390 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.559464 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.559520 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.559665 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.559709 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20260926 06:57:19.560206 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.556091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:40561 (local address 127.0.0.1:60020)
0926 06:57:19.556548 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.556557 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.556605 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.556822 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.556823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.556825 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:19.556868 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.556869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.557750 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.557751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.558940 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.558943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.559981 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.559982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.560044 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.560048 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.560107 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.560142 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":5,"thread_start_us":77,"threads_started":1}
I20260926 06:57:19.560271 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.556189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.1.74.67:40561)
0926 06:57:19.556535 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.556539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.556597 (+ 58us) server_negotiation.cc:407] Connection header received
0926 06:57:19.556669 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.556670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.556684 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.556758 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:19.556949 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.557647 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.559030 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.559823 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.559893 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.559949 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.560137 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.560177 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":101,"threads_started":1}
I20260926 06:57:19.560689 2571 tablet_service.cc:1467] Processing CreateTablet for tablet 10bf9e2e5110440184a69dcbb2b67662 (DEFAULT_TABLE table=test-table [id=89e9e86ad8e34bca9d0802b216351f30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:19.561079 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10bf9e2e5110440184a69dcbb2b67662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:19.561040 2851 tablet_service.cc:1467] Processing CreateTablet for tablet 10bf9e2e5110440184a69dcbb2b67662 (DEFAULT_TABLE table=test-table [id=89e9e86ad8e34bca9d0802b216351f30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:19.561367 2851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10bf9e2e5110440184a69dcbb2b67662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:19.561899 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.556103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:38171 (local address 127.0.0.1:46402)
0926 06:57:19.556689 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.556697 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.556732 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.557949 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.557950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.557952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:19.558001 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.558002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.559181 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.559182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.560705 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.560707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.561660 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.561661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.561721 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.561725 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.561794 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.561833 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":88,"threads_started":1}
I20260926 06:57:19.561931 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.556390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.1.74.66:38171)
0926 06:57:19.557745 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.557749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.557759 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:19.557795 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.557797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.557812 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.557885 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:19.558076 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.559106 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.560890 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.561573 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.561649 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.561695 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.561806 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.561843 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1294,"thread_start_us":90,"threads_started":1}
I20260926 06:57:19.564235 2940 tablet_bootstrap.cc:492] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: Bootstrap starting.
I20260926 06:57:19.564342 2941 tablet_bootstrap.cc:492] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: Bootstrap starting.
I20260926 06:57:19.564707 2711 tablet_service.cc:1467] Processing CreateTablet for tablet 10bf9e2e5110440184a69dcbb2b67662 (DEFAULT_TABLE table=test-table [id=89e9e86ad8e34bca9d0802b216351f30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:19.564889 2941 tablet_bootstrap.cc:654] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:19.565078 2711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10bf9e2e5110440184a69dcbb2b67662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:19.565122 2941 log.cc:824] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:19.565423 2940 tablet_bootstrap.cc:654] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:19.565758 2941 tablet_bootstrap.cc:492] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: No bootstrap required, opened a new log
I20260926 06:57:19.565759 2940 log.cc:824] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:19.565836 2941 ts_tablet_manager.cc:1397] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:19.566486 2940 tablet_bootstrap.cc:492] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: No bootstrap required, opened a new log
I20260926 06:57:19.566555 2940 ts_tablet_manager.cc:1397] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:19.567507 2941 raft_consensus.cc:348] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } }
I20260926 06:57:19.567658 2941 raft_consensus.cc:374] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:19.567698 2941 raft_consensus.cc:729] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fcc9a8cdfe491f971ddcbad355bdce, State: Initialized, Role: FOLLOWER
I20260926 06:57:19.567811 2941 consensus_queue.cc:260] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } }
I20260926 06:57:19.568077 2941 ts_tablet_manager.cc:1428] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:19.568230 2637 heartbeater.cc:503] Master 127.1.74.126:33881 was elected leader, sending a full tablet report...
I20260926 06:57:19.568645 2945 tablet_bootstrap.cc:492] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: Bootstrap starting.
I20260926 06:57:19.568683 2940 raft_consensus.cc:348] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } }
I20260926 06:57:19.568837 2940 raft_consensus.cc:374] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:19.568902 2940 raft_consensus.cc:729] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf3cb095ee5469fa75c149a9277de7a, State: Initialized, Role: FOLLOWER
I20260926 06:57:19.569010 2940 consensus_queue.cc:260] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } }
I20260926 06:57:19.569274 2940 ts_tablet_manager.cc:1428] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:57:19.569280 2917 heartbeater.cc:503] Master 127.1.74.126:33881 was elected leader, sending a full tablet report...
I20260926 06:57:19.569382 2945 tablet_bootstrap.cc:654] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:19.569707 2945 log.cc:824] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:19.570418 2945 tablet_bootstrap.cc:492] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: No bootstrap required, opened a new log
I20260926 06:57:19.570472 2945 ts_tablet_manager.cc:1397] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:19.571924 2945 raft_consensus.cc:348] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } }
I20260926 06:57:19.572048 2945 raft_consensus.cc:374] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:19.572069 2945 raft_consensus.cc:729] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7787eed3cf464f5c867f8e66bed92dd6, State: Initialized, Role: FOLLOWER
I20260926 06:57:19.572168 2945 consensus_queue.cc:260] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } }
I20260926 06:57:19.572424 2945 ts_tablet_manager.cc:1428] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:19.572459 2777 heartbeater.cc:503] Master 127.1.74.126:33881 was elected leader, sending a full tablet report...
W20260926 06:57:19.600616 2778 tablet.cc:2369] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:57:19.696427 2638 tablet.cc:2369] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:19.709668 2946 raft_consensus.cc:482] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:19.709820 2946 raft_consensus.cc:504] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } }
I20260926 06:57:19.710590 2946 leader_election.cc:288] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7fcc9a8cdfe491f971ddcbad355bdce (127.1.74.65:33965), 7787eed3cf464f5c867f8e66bed92dd6 (127.1.74.66:38171)
I20260926 06:57:19.713749 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.710399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:38171 (local address 127.1.74.67:48269)
0926 06:57:19.710452 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.710463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.710514 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.710672 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.710673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.710675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:19.710748 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.710750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.711654 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.711655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.712765 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.712768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.713524 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.713525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.713594 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.713597 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.713641 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.713677 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 06:57:19.713999 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.710447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:48269 (local address 127.1.74.66:38171)
0926 06:57:19.710493 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.710496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.710507 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:57:19.710574 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.710576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.710588 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.710669 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:19.710830 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.711562 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.712871 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.713407 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.713471 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.713493 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.713903 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.713942 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:57:19.714267 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.710593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:45525 (local address 127.1.74.65:33965)
0926 06:57:19.710730 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.710732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.710740 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:19.710763 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.710764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.710774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.710816 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:19.710977 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.711776 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.713231 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.713854 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.713922 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.713931 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.714152 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.714197 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 06:57:19.714249 2731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf9e2e5110440184a69dcbb2b67662" candidate_uuid: "adf3cb095ee5469fa75c149a9277de7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7787eed3cf464f5c867f8e66bed92dd6" is_pre_election: true
I20260926 06:57:19.714382 2731 raft_consensus.cc:2454] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adf3cb095ee5469fa75c149a9277de7a in term 0.
I20260926 06:57:19.714491 2949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.710408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:33965 (local address 127.1.74.67:45525)
0926 06:57:19.710617 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.710627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.710672 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.710840 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.710841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.710843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:19.710918 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.710920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.711871 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.711872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.713136 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.713140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.713973 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.713974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.714048 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.714052 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.714359 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.714407 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260926 06:57:19.714615 2803 leader_election.cc:302] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7787eed3cf464f5c867f8e66bed92dd6, adf3cb095ee5469fa75c149a9277de7a; no voters:
I20260926 06:57:19.714823 2946 raft_consensus.cc:2781] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:19.714836 2591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf9e2e5110440184a69dcbb2b67662" candidate_uuid: "adf3cb095ee5469fa75c149a9277de7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" is_pre_election: true
I20260926 06:57:19.714898 2946 raft_consensus.cc:482] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:19.714937 2946 raft_consensus.cc:3037] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:19.714974 2591 raft_consensus.cc:2454] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adf3cb095ee5469fa75c149a9277de7a in term 0.
I20260926 06:57:19.716221 2946 raft_consensus.cc:504] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } }
I20260926 06:57:19.716490 2591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf9e2e5110440184a69dcbb2b67662" candidate_uuid: "adf3cb095ee5469fa75c149a9277de7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce"
I20260926 06:57:19.716518 2946 leader_election.cc:288] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [CANDIDATE]: Term 1 election: Requested vote from peers f7fcc9a8cdfe491f971ddcbad355bdce (127.1.74.65:33965), 7787eed3cf464f5c867f8e66bed92dd6 (127.1.74.66:38171)
I20260926 06:57:19.716540 2731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10bf9e2e5110440184a69dcbb2b67662" candidate_uuid: "adf3cb095ee5469fa75c149a9277de7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7787eed3cf464f5c867f8e66bed92dd6"
I20260926 06:57:19.716568 2591 raft_consensus.cc:3037] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:19.716619 2731 raft_consensus.cc:3037] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:19.717415 2731 raft_consensus.cc:2454] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adf3cb095ee5469fa75c149a9277de7a in term 1.
I20260926 06:57:19.717415 2591 raft_consensus.cc:2454] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adf3cb095ee5469fa75c149a9277de7a in term 1.
I20260926 06:57:19.717597 2805 leader_election.cc:302] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adf3cb095ee5469fa75c149a9277de7a, f7fcc9a8cdfe491f971ddcbad355bdce; no voters:
I20260926 06:57:19.717708 2946 raft_consensus.cc:2781] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:19.717859 2946 raft_consensus.cc:686] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 1 LEADER]: Becoming Leader. State: Replica: adf3cb095ee5469fa75c149a9277de7a, State: Running, Role: LEADER
I20260926 06:57:19.718169 2946 consensus_queue.cc:237] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } }
I20260926 06:57:19.719130 2434 catalog_manager.cc:5495] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a reported cstate change: term changed from 0 to 1, leader changed from <none> to adf3cb095ee5469fa75c149a9277de7a (127.1.74.67). New cstate: current_term: 1 leader_uuid: "adf3cb095ee5469fa75c149a9277de7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } health_report { overall_health: UNKNOWN } } }
I20260926 06:57:19.742059 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.739547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.1.74.67:40561)
0926 06:57:19.739592 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.739594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.739623 (+ 29us) server_negotiation.cc:407] Connection header received
0926 06:57:19.739712 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.739713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.739730 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.739790 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:19.739900 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.740357 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.741248 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.741427 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.741482 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.741707 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:19.741792 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:19.741849 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.741930 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.741946 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:57:19.743762 2731 raft_consensus.cc:1261] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Refusing update from remote peer adf3cb095ee5469fa75c149a9277de7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:19.743780 2591 raft_consensus.cc:1261] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Refusing update from remote peer adf3cb095ee5469fa75c149a9277de7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:19.744094 2950 consensus_queue.cc:1035] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:19.744256 2946 consensus_queue.cc:1035] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:19.746363 2953 mvcc.cc:205] Tried to move back new op lower bound from 7333502319587651584 to 7333502319486672896. Current Snapshot: MvccSnapshot[applied={T|T < 7333502319587651584}]
I20260926 06:57:19.746865 2954 mvcc.cc:205] Tried to move back new op lower bound from 7333502319587651584 to 7333502319486672896. Current Snapshot: MvccSnapshot[applied={T|T < 7333502319587651584 or (T in {7333502319587651584})}]
I20260926 06:57:19.749915 2433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:19.750020 2433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:19.750702 2433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05: Sending DeleteTablet for 3 replicas of tablet 10bf9e2e5110440184a69dcbb2b67662
I20260926 06:57:19.751024 2711 tablet_service.cc:1514] Processing DeleteTablet for tablet 10bf9e2e5110440184a69dcbb2b67662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:19 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:46402
I20260926 06:57:19.751026 2851 tablet_service.cc:1514] Processing DeleteTablet for tablet 10bf9e2e5110440184a69dcbb2b67662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:19 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:60020
I20260926 06:57:19.751060 2571 tablet_service.cc:1514] Processing DeleteTablet for tablet 10bf9e2e5110440184a69dcbb2b67662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:19 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:43504
I20260926 06:57:19.751276 2965 tablet_replica.cc:330] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: stopping tablet replica
I20260926 06:57:19.751371 2965 raft_consensus.cc:2229] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:19.751477 2965 raft_consensus.cc:2258] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:19.751507 2433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:19.751643 2433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:19.751927 2963 tablet_replica.cc:330] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: stopping tablet replica
I20260926 06:57:19.752027 2963 raft_consensus.cc:2229] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:19.751928 2965 ts_tablet_manager.cc:1905] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:19.753329 2711 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4f383b7611492489b9e9e10464dd83 (DEFAULT_TABLE table=test-table [id=ba18f7d4768444a4a8374fb9fa3a7894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:19.753458 2711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4f383b7611492489b9e9e10464dd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:19.754030 2965 ts_tablet_manager.cc:1918] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:19.754102 2965 log.cc:1196] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/10bf9e2e5110440184a69dcbb2b67662
I20260926 06:57:19.754362 2965 ts_tablet_manager.cc:1939] T 10bf9e2e5110440184a69dcbb2b67662 P f7fcc9a8cdfe491f971ddcbad355bdce: Deleting consensus metadata
I20260926 06:57:19.754954 2945 tablet_bootstrap.cc:492] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: Bootstrap starting.
I20260926 06:57:19.755389 2945 tablet_bootstrap.cc:654] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:19.755422 2421 catalog_manager.cc:4841] TS f7fcc9a8cdfe491f971ddcbad355bdce (127.1.74.65:33965): tablet 10bf9e2e5110440184a69dcbb2b67662 (table test-table [id=89e9e86ad8e34bca9d0802b216351f30]) successfully deleted
I20260926 06:57:19.755499 2571 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4f383b7611492489b9e9e10464dd83 (DEFAULT_TABLE table=test-table [id=ba18f7d4768444a4a8374fb9fa3a7894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:19.755595 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4f383b7611492489b9e9e10464dd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:19.755751 2963 raft_consensus.cc:2258] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:19.755952 2964 tablet_replica.cc:330] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: stopping tablet replica
I20260926 06:57:19.755970 2945 tablet_bootstrap.cc:492] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: No bootstrap required, opened a new log
I20260926 06:57:19.756084 2945 ts_tablet_manager.cc:1397] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:19.756196 2964 raft_consensus.cc:2229] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:19.756278 2963 ts_tablet_manager.cc:1905] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:19.756246 2945 raft_consensus.cc:348] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.756501 2945 raft_consensus.cc:374] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:19.756541 2945 raft_consensus.cc:729] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7787eed3cf464f5c867f8e66bed92dd6, State: Initialized, Role: FOLLOWER
I20260926 06:57:19.756585 2945 consensus_queue.cc:260] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.756625 2964 raft_consensus.cc:2258] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:19.756680 2945 ts_tablet_manager.cc:1428] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:19.756946 2941 tablet_bootstrap.cc:492] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: Bootstrap starting.
I20260926 06:57:19.757256 2941 tablet_bootstrap.cc:654] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:19.757262 2964 ts_tablet_manager.cc:1905] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:19.755023 2851 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4f383b7611492489b9e9e10464dd83 (DEFAULT_TABLE table=test-table [id=ba18f7d4768444a4a8374fb9fa3a7894]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:19.757604 2851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d4f383b7611492489b9e9e10464dd83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:19.757918 2963 ts_tablet_manager.cc:1918] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:19.757984 2963 log.cc:1196] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/10bf9e2e5110440184a69dcbb2b67662
I20260926 06:57:19.758250 2963 ts_tablet_manager.cc:1939] T 10bf9e2e5110440184a69dcbb2b67662 P adf3cb095ee5469fa75c149a9277de7a: Deleting consensus metadata
I20260926 06:57:19.758826 2940 tablet_bootstrap.cc:492] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: Bootstrap starting.
I20260926 06:57:19.758944 2418 catalog_manager.cc:4841] TS adf3cb095ee5469fa75c149a9277de7a (127.1.74.67:40561): tablet 10bf9e2e5110440184a69dcbb2b67662 (table test-table [id=89e9e86ad8e34bca9d0802b216351f30]) successfully deleted
I20260926 06:57:19.759121 2964 ts_tablet_manager.cc:1918] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:19.759209 2940 tablet_bootstrap.cc:654] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:19.759227 2964 log.cc:1196] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/10bf9e2e5110440184a69dcbb2b67662
I20260926 06:57:19.759598 2964 ts_tablet_manager.cc:1939] T 10bf9e2e5110440184a69dcbb2b67662 P 7787eed3cf464f5c867f8e66bed92dd6: Deleting consensus metadata
I20260926 06:57:19.759850 2941 tablet_bootstrap.cc:492] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: No bootstrap required, opened a new log
I20260926 06:57:19.759897 2941 ts_tablet_manager.cc:1397] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:57:19.760041 2941 raft_consensus.cc:348] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.760105 2941 raft_consensus.cc:374] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:19.760124 2941 raft_consensus.cc:729] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fcc9a8cdfe491f971ddcbad355bdce, State: Initialized, Role: FOLLOWER
I20260926 06:57:19.760106 2419 catalog_manager.cc:4841] TS 7787eed3cf464f5c867f8e66bed92dd6 (127.1.74.66:38171): tablet 10bf9e2e5110440184a69dcbb2b67662 (table test-table [id=89e9e86ad8e34bca9d0802b216351f30]) successfully deleted
I20260926 06:57:19.760186 2941 consensus_queue.cc:260] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.760309 2941 ts_tablet_manager.cc:1428] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.761154 2940 tablet_bootstrap.cc:492] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: No bootstrap required, opened a new log
I20260926 06:57:19.761207 2940 ts_tablet_manager.cc:1397] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:19.761348 2940 raft_consensus.cc:348] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.761443 2940 raft_consensus.cc:374] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:19.761466 2940 raft_consensus.cc:729] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf3cb095ee5469fa75c149a9277de7a, State: Initialized, Role: FOLLOWER
I20260926 06:57:19.761523 2940 consensus_queue.cc:260] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.761626 2940 ts_tablet_manager.cc:1428] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260926 06:57:19.766431 2918 tablet.cc:2369] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:19.837033 2958 raft_consensus.cc:482] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:19.837162 2958 raft_consensus.cc:504] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.837380 2958 leader_election.cc:288] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7fcc9a8cdfe491f971ddcbad355bdce (127.1.74.65:33965), 7787eed3cf464f5c867f8e66bed92dd6 (127.1.74.66:38171)
I20260926 06:57:19.837590 2591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4f383b7611492489b9e9e10464dd83" candidate_uuid: "adf3cb095ee5469fa75c149a9277de7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" is_pre_election: true
I20260926 06:57:19.837630 2731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4f383b7611492489b9e9e10464dd83" candidate_uuid: "adf3cb095ee5469fa75c149a9277de7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7787eed3cf464f5c867f8e66bed92dd6" is_pre_election: true
I20260926 06:57:19.837735 2591 raft_consensus.cc:2454] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adf3cb095ee5469fa75c149a9277de7a in term 0.
I20260926 06:57:19.837755 2731 raft_consensus.cc:2454] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adf3cb095ee5469fa75c149a9277de7a in term 0.
I20260926 06:57:19.837903 2805 leader_election.cc:302] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adf3cb095ee5469fa75c149a9277de7a, f7fcc9a8cdfe491f971ddcbad355bdce; no voters:
I20260926 06:57:19.837976 2958 raft_consensus.cc:2781] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:19.838011 2958 raft_consensus.cc:482] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:19.838034 2958 raft_consensus.cc:3037] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:19.838742 2958 raft_consensus.cc:504] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.838888 2958 leader_election.cc:288] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [CANDIDATE]: Term 1 election: Requested vote from peers f7fcc9a8cdfe491f971ddcbad355bdce (127.1.74.65:33965), 7787eed3cf464f5c867f8e66bed92dd6 (127.1.74.66:38171)
I20260926 06:57:19.839025 2731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4f383b7611492489b9e9e10464dd83" candidate_uuid: "adf3cb095ee5469fa75c149a9277de7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7787eed3cf464f5c867f8e66bed92dd6"
I20260926 06:57:19.839094 2731 raft_consensus.cc:3037] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:19.839074 2591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d4f383b7611492489b9e9e10464dd83" candidate_uuid: "adf3cb095ee5469fa75c149a9277de7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce"
I20260926 06:57:19.839567 2591 raft_consensus.cc:3037] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:19.839867 2731 raft_consensus.cc:2454] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adf3cb095ee5469fa75c149a9277de7a in term 1.
I20260926 06:57:19.840031 2803 leader_election.cc:302] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7787eed3cf464f5c867f8e66bed92dd6, adf3cb095ee5469fa75c149a9277de7a; no voters:
I20260926 06:57:19.840127 2958 raft_consensus.cc:2781] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:19.840170 2958 raft_consensus.cc:686] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 1 LEADER]: Becoming Leader. State: Replica: adf3cb095ee5469fa75c149a9277de7a, State: Running, Role: LEADER
I20260926 06:57:19.840224 2958 consensus_queue.cc:237] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.840773 2591 raft_consensus.cc:2454] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adf3cb095ee5469fa75c149a9277de7a in term 1.
I20260926 06:57:19.840759 2433 catalog_manager.cc:5495] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a reported cstate change: term changed from 0 to 1, leader changed from <none> to adf3cb095ee5469fa75c149a9277de7a (127.1.74.67). New cstate: current_term: 1 leader_uuid: "adf3cb095ee5469fa75c149a9277de7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } health_report { overall_health: HEALTHY } } }
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:19.886013 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.875959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33881 (local address 127.0.0.1:32892)
0926 06:57:19.876197 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.876209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.876313 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.876489 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.876490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.876813 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:19.876857 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.876860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.877568 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.877569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.878103 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.878104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.878471 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.878472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.878525 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.882106 (+ 3581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:19.882123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:19.883553 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:19.884837 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.884843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.884845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.885101 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.885289 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:19.885289 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:19.885290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:19.885349 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:19.885644 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:19.885668 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:19.885719 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.885795 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.885900 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":116,"threads_started":1}
I20260926 06:57:19.886020 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.876067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.1.74.126:33881)
0926 06:57:19.876171 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.876174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.876294 (+ 120us) server_negotiation.cc:407] Connection header received
0926 06:57:19.876370 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.876371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.876381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.876481 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:19.876950 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.877464 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.878191 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.878373 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.878467 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.883643 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:19.883645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:19.883645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:19.883651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:19.884768 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.885180 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.885181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.885181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.885230 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.885416 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.885416 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.885417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.885532 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:19.885643 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.885827 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.885946 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":74}
I20260926 06:57:19.888175 2433 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:32892:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 06:57:19.888366 2433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:19.889737 2711 tablet_service.cc:1467] Processing CreateTablet for tablet fe21a1f20bb94cfb8b76d1591525e300 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=745dbdbf1afe42ee9f320ab32afb5ffb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:57:19.889761 2571 tablet_service.cc:1467] Processing CreateTablet for tablet fe21a1f20bb94cfb8b76d1591525e300 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=745dbdbf1afe42ee9f320ab32afb5ffb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:57:19.889801 2851 tablet_service.cc:1467] Processing CreateTablet for tablet fe21a1f20bb94cfb8b76d1591525e300 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=745dbdbf1afe42ee9f320ab32afb5ffb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:57:19.889916 2851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe21a1f20bb94cfb8b76d1591525e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:19.889917 2711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe21a1f20bb94cfb8b76d1591525e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:19.889921 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe21a1f20bb94cfb8b76d1591525e300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:19.891011 2941 tablet_bootstrap.cc:492] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: Bootstrap starting.
I20260926 06:57:19.891376 2940 tablet_bootstrap.cc:492] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: Bootstrap starting.
I20260926 06:57:19.891507 2941 tablet_bootstrap.cc:654] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:19.891708 2940 tablet_bootstrap.cc:654] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:19.891999 2945 tablet_bootstrap.cc:492] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: Bootstrap starting.
I20260926 06:57:19.892141 2941 tablet_bootstrap.cc:492] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: No bootstrap required, opened a new log
I20260926 06:57:19.892210 2941 ts_tablet_manager.cc:1397] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:19.892345 2945 tablet_bootstrap.cc:654] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:19.892356 2941 raft_consensus.cc:348] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.892447 2941 raft_consensus.cc:374] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:19.892474 2941 raft_consensus.cc:729] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fcc9a8cdfe491f971ddcbad355bdce, State: Initialized, Role: FOLLOWER
I20260926 06:57:19.892525 2941 consensus_queue.cc:260] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.892620 2941 ts_tablet_manager.cc:1428] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.892681 2944 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.892787 2944 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:19.892804 2940 tablet_bootstrap.cc:492] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: No bootstrap required, opened a new log
I20260926 06:57:19.892844 2940 ts_tablet_manager.cc:1397] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:19.892966 2940 raft_consensus.cc:348] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.893060 2940 raft_consensus.cc:374] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:19.893095 2940 raft_consensus.cc:729] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf3cb095ee5469fa75c149a9277de7a, State: Initialized, Role: FOLLOWER
I20260926 06:57:19.893149 2940 consensus_queue.cc:260] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.893244 2940 ts_tablet_manager.cc:1428] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:19.893356 2958 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.893431 2958 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:19.893746 2945 tablet_bootstrap.cc:492] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: No bootstrap required, opened a new log
I20260926 06:57:19.893795 2945 ts_tablet_manager.cc:1397] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:19.893935 2945 raft_consensus.cc:348] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.893985 2945 raft_consensus.cc:374] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:19.894032 2945 raft_consensus.cc:729] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7787eed3cf464f5c867f8e66bed92dd6, State: Initialized, Role: FOLLOWER
I20260926 06:57:19.894085 2945 consensus_queue.cc:260] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.894189 2945 ts_tablet_manager.cc:1428] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:19.894223 2948 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.894411 2948 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:19.930871 2944 raft_consensus.cc:482] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:19.930955 2944 raft_consensus.cc:504] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.931325 2944 leader_election.cc:288] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7787eed3cf464f5c867f8e66bed92dd6 (127.1.74.66:38171), adf3cb095ee5469fa75c149a9277de7a (127.1.74.67:40561)
I20260926 06:57:19.935829 2979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.931514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:38171 (local address 127.1.74.65:57623)
0926 06:57:19.931710 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.931720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.931793 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.931935 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.931935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.931937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:19.932021 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.932023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.932951 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.932952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.934326 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.934329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.935579 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.935580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.935655 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.935659 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.935710 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.935735 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":85,"thread_start_us":111,"threads_started":1}
I20260926 06:57:19.935854 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.931629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:41529 (local address 127.1.74.67:40561)
0926 06:57:19.931681 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.931683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.931794 (+ 111us) server_negotiation.cc:407] Connection header received
0926 06:57:19.931862 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.931862 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.931872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.931923 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:19.932095 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.932864 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.934414 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.934972 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.935038 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.935048 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.935740 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.935783 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:57:19.935854 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.931562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:57623 (local address 127.1.74.66:38171)
0926 06:57:19.931679 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.931681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.931794 (+ 113us) server_negotiation.cc:407] Connection header received
0926 06:57:19.931852 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.931853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.931864 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.931923 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:19.932095 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.932863 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.934414 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.934962 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.935038 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.935048 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.935740 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.935784 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 06:57:19.935832 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.931508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:40561 (local address 127.1.74.65:41529)
0926 06:57:19.931707 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.931720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.931793 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.931935 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.931935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.931939 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:19.932023 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.932024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.932944 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.932945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.934329 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.934331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.935073 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.935074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.935602 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.935606 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.935669 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.935720 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":72,"thread_start_us":112,"threads_started":1}
I20260926 06:57:19.936241 2731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe21a1f20bb94cfb8b76d1591525e300" candidate_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7787eed3cf464f5c867f8e66bed92dd6" is_pre_election: true
I20260926 06:57:19.936334 2731 raft_consensus.cc:2454] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7fcc9a8cdfe491f971ddcbad355bdce in term 0.
I20260926 06:57:19.936345 2871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe21a1f20bb94cfb8b76d1591525e300" candidate_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf3cb095ee5469fa75c149a9277de7a" is_pre_election: true
I20260926 06:57:19.936502 2871 raft_consensus.cc:2454] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7fcc9a8cdfe491f971ddcbad355bdce in term 0.
I20260926 06:57:19.936501 2523 leader_election.cc:302] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7787eed3cf464f5c867f8e66bed92dd6, f7fcc9a8cdfe491f971ddcbad355bdce; no voters:
I20260926 06:57:19.936981 2944 raft_consensus.cc:2781] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:19.937031 2944 raft_consensus.cc:482] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:19.937045 2944 raft_consensus.cc:3037] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:19.937747 2944 raft_consensus.cc:504] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.937894 2944 leader_election.cc:288] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [CANDIDATE]: Term 1 election: Requested vote from peers 7787eed3cf464f5c867f8e66bed92dd6 (127.1.74.66:38171), adf3cb095ee5469fa75c149a9277de7a (127.1.74.67:40561)
I20260926 06:57:19.938071 2871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe21a1f20bb94cfb8b76d1591525e300" candidate_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf3cb095ee5469fa75c149a9277de7a"
I20260926 06:57:19.938091 2731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe21a1f20bb94cfb8b76d1591525e300" candidate_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7787eed3cf464f5c867f8e66bed92dd6"
I20260926 06:57:19.938156 2871 raft_consensus.cc:3037] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:19.938185 2731 raft_consensus.cc:3037] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:19.938763 2731 raft_consensus.cc:2454] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7fcc9a8cdfe491f971ddcbad355bdce in term 1.
I20260926 06:57:19.938930 2523 leader_election.cc:302] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7787eed3cf464f5c867f8e66bed92dd6, f7fcc9a8cdfe491f971ddcbad355bdce; no voters:
I20260926 06:57:19.938944 2871 raft_consensus.cc:2454] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7fcc9a8cdfe491f971ddcbad355bdce in term 1.
I20260926 06:57:19.939023 2944 raft_consensus.cc:2781] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:19.939167 2944 raft_consensus.cc:686] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 LEADER]: Becoming Leader. State: Replica: f7fcc9a8cdfe491f971ddcbad355bdce, State: Running, Role: LEADER
I20260926 06:57:19.939247 2944 consensus_queue.cc:237] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } }
I20260926 06:57:19.939775 2981 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: TxnStatusTablet state changed. Reason: New leader f7fcc9a8cdfe491f971ddcbad355bdce. Latest consensus state: current_term: 1 leader_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.939869 2981 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:57:19.939919 2434 catalog_manager.cc:5495] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce reported cstate change: term changed from 0 to 1, leader changed from <none> to f7fcc9a8cdfe491f971ddcbad355bdce (127.1.74.65). New cstate: current_term: 1 leader_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } health_report { overall_health: UNKNOWN } } }
I20260926 06:57:19.940083 2983 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 06:57:19.940186 2983 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 06:57:19.956599 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.953758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.1.74.65:33965)
0926 06:57:19.953798 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.953800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.953830 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:57:19.953927 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.953928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.953943 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.954002 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:19.954155 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.954639 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.955809 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.956059 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.956112 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.956307 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:19.956401 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:19.956451 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.956522 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.956539 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:57:19.956651 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.953698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:33965 (local address 127.0.0.1:43518)
0926 06:57:19.953783 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.953795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.953849 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.954025 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.954026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.954027 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:19.954081 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.954084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.954705 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.954706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.955735 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.955738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.956125 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.956126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.956194 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.956213 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:19.956456 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:19.956464 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.956520 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.956568 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 06:57:19.957778 2731 raft_consensus.cc:1261] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Refusing update from remote peer f7fcc9a8cdfe491f971ddcbad355bdce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:19.957940 2871 raft_consensus.cc:1261] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a [term 1 FOLLOWER]: Refusing update from remote peer f7fcc9a8cdfe491f971ddcbad355bdce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:19.958242 2981 consensus_queue.cc:1035] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:19.958362 2944 consensus_queue.cc:1035] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:19.959020 2948 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: TxnStatusTablet state changed. Reason: New leader f7fcc9a8cdfe491f971ddcbad355bdce. Latest consensus state: current_term: 1 leader_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.959040 2958 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: TxnStatusTablet state changed. Reason: New leader f7fcc9a8cdfe491f971ddcbad355bdce. Latest consensus state: current_term: 1 leader_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.959112 2958 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:19.959112 2948 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:19.959986 2944 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.960071 2944 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:57:19.960126 2944 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.960162 2944 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P f7fcc9a8cdfe491f971ddcbad355bdce: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:57:19.960209 2955 mvcc.cc:205] Tried to move back new op lower bound from 7333502320464904192 to 7333502320391725056. Current Snapshot: MvccSnapshot[applied={T|T < 7333502320464904192 or (T in {7333502320464904192})}]
I20260926 06:57:19.960188 2948 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.960686 2948 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P 7787eed3cf464f5c867f8e66bed92dd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:19.961315 2958 tablet_replica.cc:439] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 } } peers { permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 } } peers { permanent_uuid: "adf3cb095ee5469fa75c149a9277de7a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40561 } } }
I20260926 06:57:19.961416 2958 tablet_replica.cc:442] T fe21a1f20bb94cfb8b76d1591525e300 P adf3cb095ee5469fa75c149a9277de7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:19.965965 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.963671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:33965 (local address 127.1.74.67:59831)
0926 06:57:19.963746 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.963759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.963817 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.963941 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.963942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.963944 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:19.963989 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.963991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.964577 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.964578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.965156 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.965158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.965537 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.965538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.965594 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.965609 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:19.965797 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:19.965800 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.965847 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.965898 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 06:57:19.966583 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.963729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:59831 (local address 127.1.74.65:33965)
0926 06:57:19.963774 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.963775 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.963799 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:57:19.963863 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.963864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.963872 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.963938 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:19.964041 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.964500 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.965243 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.965441 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.965539 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.965701 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:19.965739 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:19.965797 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.966521 (+ 724us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.966537 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:57:19.968643 2591 raft_consensus.cc:1261] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Refusing update from remote peer adf3cb095ee5469fa75c149a9277de7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:19.968650 2731 raft_consensus.cc:1261] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Refusing update from remote peer adf3cb095ee5469fa75c149a9277de7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:19.968856 2946 consensus_queue.cc:1035] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7787eed3cf464f5c867f8e66bed92dd6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:19.968956 2958 consensus_queue.cc:1035] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fcc9a8cdfe491f971ddcbad355bdce" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 33965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:19.977008 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.973947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:59775 (local address 127.1.74.67:40561)
0926 06:57:19.973984 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.973986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.974118 (+ 132us) server_negotiation.cc:407] Connection header received
0926 06:57:19.974190 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.974191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.974202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.974265 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:19.974386 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.975115 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.976303 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.976508 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.976589 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.976779 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:19.976811 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:19.976858 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:19.976923 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 06:57:19.976942 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:57:19.977048 2994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.973907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:40561 (local address 127.1.74.65:59775)
0926 06:57:19.974053 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:19.974066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:19.974137 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:19.974276 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:19.974276 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:19.974278 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:19.974326 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.974327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.975187 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.975187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.976227 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:19.976229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.976591 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.976592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:19.976675 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.976688 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:19.976867 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:19.976868 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:57:19.976918 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:57:19.976956 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260926 06:57:19.985095 2433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:19.985215 2433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:19.985864 2433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8d0af46cf334dfda545c2c18cbe1b05: Sending DeleteTablet for 3 replicas of tablet 8d4f383b7611492489b9e9e10464dd83
I20260926 06:57:19.986100 2571 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4f383b7611492489b9e9e10464dd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:19 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:43504
I20260926 06:57:19.986135 2711 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4f383b7611492489b9e9e10464dd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:19 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:46402
I20260926 06:57:19.986208 2965 tablet_replica.cc:330] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: stopping tablet replica
I20260926 06:57:19.986254 2964 tablet_replica.cc:330] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: stopping tablet replica
I20260926 06:57:19.986280 2965 raft_consensus.cc:2229] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:19.986303 2964 raft_consensus.cc:2229] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:19.986315 2965 raft_consensus.cc:2258] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:19.986346 2964 raft_consensus.cc:2258] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:19.986704 2965 ts_tablet_manager.cc:1905] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:19.986704 2851 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4f383b7611492489b9e9e10464dd83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:19 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:60020
I20260926 06:57:19.986793 2963 tablet_replica.cc:330] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: stopping tablet replica
I20260926 06:57:19.986836 2963 raft_consensus.cc:2229] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:19.986891 2963 raft_consensus.cc:2258] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:19.987198 2963 ts_tablet_manager.cc:1905] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:19.988143 2964 ts_tablet_manager.cc:1905] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:19.988725 2963 ts_tablet_manager.cc:1918] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:19.988761 2965 ts_tablet_manager.cc:1918] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:19.988798 2963 log.cc:1196] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/8d4f383b7611492489b9e9e10464dd83
I20260926 06:57:19.988811 2965 log.cc:1196] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/8d4f383b7611492489b9e9e10464dd83
I20260926 06:57:19.989027 2963 ts_tablet_manager.cc:1939] T 8d4f383b7611492489b9e9e10464dd83 P adf3cb095ee5469fa75c149a9277de7a: Deleting consensus metadata
I20260926 06:57:19.989058 2965 ts_tablet_manager.cc:1939] T 8d4f383b7611492489b9e9e10464dd83 P f7fcc9a8cdfe491f971ddcbad355bdce: Deleting consensus metadata
I20260926 06:57:19.989337 2418 catalog_manager.cc:4841] TS adf3cb095ee5469fa75c149a9277de7a (127.1.74.67:40561): tablet 8d4f383b7611492489b9e9e10464dd83 (table test-table [id=ba18f7d4768444a4a8374fb9fa3a7894]) successfully deleted
I20260926 06:57:19.989372 2421 catalog_manager.cc:4841] TS f7fcc9a8cdfe491f971ddcbad355bdce (127.1.74.65:33965): tablet 8d4f383b7611492489b9e9e10464dd83 (table test-table [id=ba18f7d4768444a4a8374fb9fa3a7894]) successfully deleted
I20260926 06:57:19.989774 2964 ts_tablet_manager.cc:1918] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:19.989835 2964 log.cc:1196] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/8d4f383b7611492489b9e9e10464dd83
I20260926 06:57:19.990043 2964 ts_tablet_manager.cc:1939] T 8d4f383b7611492489b9e9e10464dd83 P 7787eed3cf464f5c867f8e66bed92dd6: Deleting consensus metadata
I20260926 06:57:19.990331 2419 catalog_manager.cc:4841] TS 7787eed3cf464f5c867f8e66bed92dd6 (127.1.74.66:38171): tablet 8d4f383b7611492489b9e9e10464dd83 (table test-table [id=ba18f7d4768444a4a8374fb9fa3a7894]) successfully deleted
Sep 26 06:57:19 dist-test-slave-sfvt krb5kdc[2389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:20.000326 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:19.991521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.1.74.65:33965)
0926 06:57:19.991568 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:19.991571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:19.991681 (+ 110us) server_negotiation.cc:407] Connection header received
0926 06:57:19.991749 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:19.991750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:19.991758 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:19.991798 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:19.992472 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.992916 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.993959 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:19.994137 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:19.994184 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:19.997346 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:19.997354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:19.997354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:19.997362 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:19.999050 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.999496 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.999500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.999500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.999558 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:19.999762 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:19.999763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:19.999763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:19.999902 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:20.000070 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:20.000140 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 06:57:20.000216 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 06:57:20.000566 2622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43528
I20260926 06:57:20.007707 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:20.001407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.1.74.126:33881)
0926 06:57:20.001453 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:20.001455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:20.001576 (+ 121us) server_negotiation.cc:407] Connection header received
0926 06:57:20.001607 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:20.001608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:20.001619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:20.001680 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:20.002122 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:20.002816 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:20.003545 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:20.003880 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:20.003949 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:20.005105 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:20.005107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:20.005107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:20.005114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:20.006590 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:20.006898 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:20.006899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:20.006900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:20.006960 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:20.007146 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:20.007147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:20.007147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:20.007337 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:20.007447 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:20.007530 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 06:57:20.007604 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 06:57:20.007889 2433 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32894
I20260926 06:57:20.008507 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 2506
I20260926 06:57:20.014588 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 2646
I20260926 06:57:20.019438 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 2786
I20260926 06:57:20.025128 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 2402
2026-09-26T06:57:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2328 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:20 dist-test-slave-sfvt krb5kdc[3002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:20 dist-test-slave-sfvt krb5kdc[3002](info): set up 2 sockets
Sep 26 06:57:20 dist-test-slave-sfvt krb5kdc[3002](info): commencing operation
krb5kdc: starting...
W20260926 06:57:21.055114 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:21.174710 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:41225
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:41225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:21.254577 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:21.254837 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:21.254873 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:21.256045 3015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:21.256122 3015 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 06:57:21.256145 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:21.256166 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:21.256197 3015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:21.256222 3015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:21.257691 3015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:41225
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:21.257982 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:21.258214 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:21.262974 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:21.262985 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:21.262982 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:21.263209 3015 server_base.cc:1034] running on GCE node
I20260926 06:57:21.263460 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:21.263782 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:21.264954 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790405841264940 us; error 43 us; skew 500 ppm
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:21.267438 3015 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:21.267833 3015 webserver.cc:460] Webserver started at http://127.1.74.126:43725/ using document root <none> and password file <none>
I20260926 06:57:21.268075 3015 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:21.268122 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:21.268270 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:21.269392 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "21aff96afdf445e1b628b50d61146aa6"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "9c1c073f2b72ed10905f1227ce2e5f24"
server_key_iv: "70d3cbe975886bafdeae20948a743745"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.269762 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "21aff96afdf445e1b628b50d61146aa6"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "9c1c073f2b72ed10905f1227ce2e5f24"
server_key_iv: "70d3cbe975886bafdeae20948a743745"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.271173 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:57:21.272109 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.272311 3015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:21.272419 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "21aff96afdf445e1b628b50d61146aa6"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "9c1c073f2b72ed10905f1227ce2e5f24"
server_key_iv: "70d3cbe975886bafdeae20948a743745"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.272505 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:21.278925 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:21.280692 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:21.280846 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:21.285090 3015 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:41225
I20260926 06:57:21.285219 3092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:41225 every 8 connection(s)
I20260926 06:57:21.285488 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:21.286159 3093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:21.288450 3093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: Bootstrap starting.
I20260926 06:57:21.289021 3093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:21.289027 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 3015
I20260926 06:57:21.289139 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:21.289307 1321 external_mini_cluster.cc:1428] Setting key b6362d150158c73aba75380de404750e
I20260926 06:57:21.289371 3093 log.cc:824] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:21.290127 3093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: No bootstrap required, opened a new log
I20260926 06:57:21.292027 3093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21aff96afdf445e1b628b50d61146aa6" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41225 } }
I20260926 06:57:21.292161 3093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:21.292183 3093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21aff96afdf445e1b628b50d61146aa6, State: Initialized, Role: FOLLOWER
I20260926 06:57:21.292315 3093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21aff96afdf445e1b628b50d61146aa6" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41225 } }
I20260926 06:57:21.292390 3093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:21.292433 3093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:21.292490 3093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:21.293071 3093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21aff96afdf445e1b628b50d61146aa6" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41225 } }
I20260926 06:57:21.293179 3093 leader_election.cc:302] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21aff96afdf445e1b628b50d61146aa6; no voters:
I20260926 06:57:21.293334 3093 leader_election.cc:288] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:21.293391 3098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:21.293532 3098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 21aff96afdf445e1b628b50d61146aa6, State: Running, Role: LEADER
I20260926 06:57:21.293565 3093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:21.293624 3098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21aff96afdf445e1b628b50d61146aa6" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41225 } }
I20260926 06:57:21.293905 3098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21aff96afdf445e1b628b50d61146aa6. Latest consensus state: current_term: 1 leader_uuid: "21aff96afdf445e1b628b50d61146aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21aff96afdf445e1b628b50d61146aa6" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41225 } } }
I20260926 06:57:21.293994 3098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:21.293915 3100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21aff96afdf445e1b628b50d61146aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21aff96afdf445e1b628b50d61146aa6" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41225 } } }
I20260926 06:57:21.294128 3100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6 [sys.catalog]: This master's current role is: LEADER
W20260926 06:57:21.295548 3114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:21.295630 3114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:21.295697 3105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:21.295902 3105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:21.297547 3105 catalog_manager.cc:1347] Generated new cluster ID: 64440b2852364a27be714019699003d1
I20260926 06:57:21.297593 3105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:21.301105 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.289774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.1.74.126:41225)
0926 06:57:21.289948 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.289964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.289989 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:57:21.290467 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.290474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.290523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.290637 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:21.291461 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.292876 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.293967 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.294649 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.294787 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.297789 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:21.297793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:21.297793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:21.297804 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:21.299595 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.299958 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.299961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.299961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.300055 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.300267 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.300268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.300268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.300497 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:21.300669 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.300764 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.300909 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260926 06:57:21.307787 3105 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:21.308473 3105 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:21.312247 3105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: Generated new TSK 0
I20260926 06:57:21.312404 3105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:21.343330 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:41225
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:21.424984 3119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:21.425196 3119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:21.425222 3119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:21.426350 3119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:21.426389 3119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:21.426410 3119 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 06:57:21.426422 3119 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:57:21.426440 3119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:21.427896 3119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.74.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:21.428086 3119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:21.428268 3119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:21.433388 3125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:21.433609 3119 server_base.cc:1034] running on GCE node
W20260926 06:57:21.433387 3127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:21.433400 3124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:21.433930 3119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:21.434135 3119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:21.435315 3119 hybrid_clock.cc:648] HybridClock initialized: now 1790405841435306 us; error 30 us; skew 500 ppm
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:21.437840 3119 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:21.438216 3119 webserver.cc:460] Webserver started at http://127.1.74.65:36253/ using document root <none> and password file <none>
I20260926 06:57:21.438395 3119 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:21.438433 3119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:21.438526 3119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:21.439622 3119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "5aefd9600b7840379c52972e0f92157c"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "44da59fc6688ddb982b0fd05a5191ed4"
server_key_iv: "dcd07ef95de6f36f30d28d456f14cb68"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.439926 3119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "5aefd9600b7840379c52972e0f92157c"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "44da59fc6688ddb982b0fd05a5191ed4"
server_key_iv: "dcd07ef95de6f36f30d28d456f14cb68"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.441172 3119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:57:21.441840 3134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.442009 3119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:21.442111 3119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "5aefd9600b7840379c52972e0f92157c"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "44da59fc6688ddb982b0fd05a5191ed4"
server_key_iv: "dcd07ef95de6f36f30d28d456f14cb68"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.442184 3119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:21.453235 3119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:21.455634 3119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:21.455808 3119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:21.456327 3119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:21.456391 3119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.456453 3119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:21.456485 3119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:21.463749 3119 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:40885
I20260926 06:57:21.463851 3249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:40885 every 8 connection(s)
I20260926 06:57:21.464188 3119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:21.466248 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.456872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:35621 (local address 127.1.74.126:41225)
0926 06:57:21.456920 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.456922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.457132 (+ 210us) server_negotiation.cc:407] Connection header received
0926 06:57:21.457726 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.457728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.457742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.457793 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:21.458675 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.459585 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.460623 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.460990 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.461074 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.463358 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:21.463360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:21.463360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:21.463368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:21.464981 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.465383 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.465383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.465384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.465433 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.465708 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.465709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.465709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.465809 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:21.465856 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.466123 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.466186 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:57:21.466506 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.456859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41225 (local address 127.1.74.65:35621)
0926 06:57:21.457013 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.457047 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.457584 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.457869 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.457872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.458268 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:21.458558 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.458564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.459685 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.459686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.460517 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.460519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.461091 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.461092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.461224 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.461754 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:21.461763 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:21.463216 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:21.465072 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.465079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.465081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.465311 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.465515 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.465516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.465516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.465641 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.465874 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:21.465880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:21.465997 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.466108 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.466240 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260926 06:57:21.467042 3250 heartbeater.cc:347] Connected to a master server at 127.1.74.126:41225
I20260926 06:57:21.467193 3250 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:21.467409 3250 heartbeater.cc:511] Master 127.1.74.126:41225 requested a full tablet report, sending...
I20260926 06:57:21.468062 3046 ts_manager.cc:194] Registered new tserver with Master: 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885)
I20260926 06:57:21.468446 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 3119
I20260926 06:57:21.468533 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:21.468703 1321 external_mini_cluster.cc:1428] Setting key 6ef073d64ca2f793a89ad72f8f3334fe
I20260926 06:57:21.469040 3046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:35621
I20260926 06:57:21.474720 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.468225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:38411 (local address 127.1.74.126:41225)
0926 06:57:21.468284 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.468286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.469004 (+ 718us) server_negotiation.cc:407] Connection header received
0926 06:57:21.469026 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.469027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.469037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.469095 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:21.469521 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.469985 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.470763 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.470948 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.471033 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.472075 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:21.472077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:21.472077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:21.472083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:21.473567 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.473940 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.473941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.473942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.473997 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.474205 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.474206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.474207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.474315 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:21.474390 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.474553 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.474623 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:57:21.474731 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.468171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41225 (local address 127.1.74.65:38411)
0926 06:57:21.468880 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.468892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.468966 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.469148 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.469149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.469374 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:21.469444 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.469446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.470079 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.470080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.470661 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.470663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.471034 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.471035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.471158 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.471688 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:21.471693 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:21.471967 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:21.473684 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.473685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.473685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.473858 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.474062 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.474063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.474063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.474125 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.474382 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:21.474383 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:21.474432 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.474512 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.474626 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:21.494606 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:41225
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:21.574743 3259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:21.574968 3259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:21.575008 3259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:21.576265 3259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:21.576334 3259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:21.576373 3259 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 06:57:21.576417 3259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:57:21.576457 3259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:21.577877 3259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.74.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:21.578131 3259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:21.578359 3259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:21.583225 3265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:21.583236 3267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:21.583417 3259 server_base.cc:1034] running on GCE node
W20260926 06:57:21.583236 3264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:21.583662 3259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:21.583889 3259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:21.585043 3259 hybrid_clock.cc:648] HybridClock initialized: now 1790405841585025 us; error 30 us; skew 500 ppm
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:21.587477 3259 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:21.587963 3259 webserver.cc:460] Webserver started at http://127.1.74.66:32969/ using document root <none> and password file <none>
I20260926 06:57:21.588172 3259 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:21.588235 3259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:21.588390 3259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:21.589357 3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "7b751efd090145f982765e75c0980b30"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "78ec4354335e053ac3d54f924f0d3c31"
server_key_iv: "92583ca969cf07b27919f1c1144fba41"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.589695 3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "7b751efd090145f982765e75c0980b30"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "78ec4354335e053ac3d54f924f0d3c31"
server_key_iv: "92583ca969cf07b27919f1c1144fba41"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.590960 3259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:21.591730 3274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.591897 3259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.591990 3259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "7b751efd090145f982765e75c0980b30"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "78ec4354335e053ac3d54f924f0d3c31"
server_key_iv: "92583ca969cf07b27919f1c1144fba41"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.592068 3259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:21.610373 3259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:21.612707 3259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:21.612874 3259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:21.613543 3259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:21.613593 3259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.613628 3259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:21.613653 3259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:21.620839 3259 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:34619
I20260926 06:57:21.620893 3389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:34619 every 8 connection(s)
I20260926 06:57:21.621246 3259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:21.622877 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.613992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:43341 (local address 127.1.74.126:41225)
0926 06:57:21.614061 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.614064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.614298 (+ 234us) server_negotiation.cc:407] Connection header received
0926 06:57:21.614752 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.614753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.614768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.614856 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:21.615646 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.616136 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.617183 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.617390 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.617501 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.620041 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:21.620043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:21.620043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:21.620052 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:21.621786 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.622127 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.622128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.622128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.622178 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.622365 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.622366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.622366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.622460 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:21.622506 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.622749 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.622805 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:57:21.623062 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.613963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41225 (local address 127.1.74.66:43341)
0926 06:57:21.614113 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.614198 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.614663 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.614966 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.614970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.615351 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:21.615529 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.615534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.616230 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.616231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.617065 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.617067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.617510 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.617511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.617629 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.618135 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:21.618143 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:21.619945 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:21.621859 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.621863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.621864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.622063 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.622217 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.622218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.622218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.622315 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.622517 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:21.622521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:21.622646 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.622725 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.622836 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 06:57:21.623610 3390 heartbeater.cc:347] Connected to a master server at 127.1.74.126:41225
I20260926 06:57:21.623714 3390 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:21.623896 3390 heartbeater.cc:511] Master 127.1.74.126:41225 requested a full tablet report, sending...
I20260926 06:57:21.624275 3047 ts_manager.cc:194] Registered new tserver with Master: 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619)
I20260926 06:57:21.624794 3047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:43341
I20260926 06:57:21.629642 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 3259
I20260926 06:57:21.629765 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:21.629980 1321 external_mini_cluster.cc:1428] Setting key 52c6697e19742f10e9ff65b86527161b
I20260926 06:57:21.630358 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.624622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:51953 (local address 127.1.74.126:41225)
0926 06:57:21.624663 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.624665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.624779 (+ 114us) server_negotiation.cc:407] Connection header received
0926 06:57:21.624841 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.624842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.624850 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.624921 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:21.625296 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.625865 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.626544 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.626725 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.626788 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.627640 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:21.627641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:21.627642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:21.627648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:21.629116 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.629383 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.629384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.629384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.629479 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.629709 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.629710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.629711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.629923 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:21.630004 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.630172 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.630259 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:57:21.630353 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.624537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41225 (local address 127.1.74.66:51953)
0926 06:57:21.624709 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.624719 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.624778 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.624939 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.624940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.625163 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:21.625225 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.625226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.625951 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.625952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.626459 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.626460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.626803 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.626804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.626863 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.627223 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:21.627227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:21.627547 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:21.629187 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.629187 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.629188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.629322 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.629570 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.629571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.629571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.629633 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.630000 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:21.630001 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:21.630060 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.630150 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.630257 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:21.657827 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:41225
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:21.738039 3399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:21.738273 3399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:21.738312 3399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:21.739480 3399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:21.739548 3399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:21.739583 3399 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 06:57:21.739614 3399 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:57:21.739645 3399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:21.741060 3399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.74.126:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:21.741324 3399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:21.741552 3399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:57:21.746537 3399 server_base.cc:1034] running on GCE node
W20260926 06:57:21.746513 3404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:21.746521 3405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:21.746505 3407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:21.746848 3399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:21.747026 3399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:21.748172 3399 hybrid_clock.cc:648] HybridClock initialized: now 1790405841748155 us; error 32 us; skew 500 ppm
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:21.750734 3399 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:21.751178 3399 webserver.cc:460] Webserver started at http://127.1.74.67:46523/ using document root <none> and password file <none>
I20260926 06:57:21.751410 3399 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:21.751456 3399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:21.751545 3399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:21.752550 3399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "9899c8b5a59741e9a704dd923f5f07b8"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "29b4be5de52aac5be6e04b2dd65fb3cc"
server_key_iv: "40290216ec63a64e689ae38045ce1ca7"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.752887 3399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "9899c8b5a59741e9a704dd923f5f07b8"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "29b4be5de52aac5be6e04b2dd65fb3cc"
server_key_iv: "40290216ec63a64e689ae38045ce1ca7"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.754098 3399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:21.754791 3414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.754963 3399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:21.755060 3399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "9899c8b5a59741e9a704dd923f5f07b8"
format_stamp: "Formatted at 2026-09-26 06:57:21 on dist-test-slave-sfvt"
server_key: "29b4be5de52aac5be6e04b2dd65fb3cc"
server_key_iv: "40290216ec63a64e689ae38045ce1ca7"
server_key_version: "encryptionkey@0"
I20260926 06:57:21.755146 3399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:21.767907 3399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:21.769726 3399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:21.769888 3399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:21.770474 3399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:21.770531 3399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.770565 3399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:21.770592 3399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:21.777910 3399 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:44219
I20260926 06:57:21.777984 3529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:44219 every 8 connection(s)
I20260926 06:57:21.778398 3399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:57:21.780210 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.770856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:40109 (local address 127.1.74.126:41225)
0926 06:57:21.770924 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.770927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.771162 (+ 235us) server_negotiation.cc:407] Connection header received
0926 06:57:21.771690 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.771691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.771705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.771788 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:21.772527 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.773368 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.774403 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.774650 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.774736 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.777187 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:21.777190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:21.777190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:21.777197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:21.778786 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.779259 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.779260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.779260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.779333 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.779585 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.779586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.779587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.779707 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:21.779783 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.780042 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.780120 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:57:21.780408 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.770855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41225 (local address 127.1.74.67:40109)
0926 06:57:21.771018 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.771056 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.771590 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.771888 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.771892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.772236 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:21.772437 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.772441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.773466 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.773467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.774284 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.774286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.774746 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.774747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.774875 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.775395 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:21.775404 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:21.777090 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:21.778890 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.778895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.778897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.779171 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.779408 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.779409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.779410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.779520 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.779775 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:21.779780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:21.779915 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.780022 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.780173 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260926 06:57:21.780858 3530 heartbeater.cc:347] Connected to a master server at 127.1.74.126:41225
I20260926 06:57:21.780958 3530 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:21.781131 3530 heartbeater.cc:511] Master 127.1.74.126:41225 requested a full tablet report, sending...
I20260926 06:57:21.781484 3046 ts_manager.cc:194] Registered new tserver with Master: 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67:44219)
I20260926 06:57:21.782027 3046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:40109
I20260926 06:57:21.782444 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 3399
I20260926 06:57:21.782543 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:21.782706 1321 external_mini_cluster.cc:1428] Setting key 039e9477cf008671ccca6107fc7599e6
I20260926 06:57:21.783530 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:57:21.787807 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.781879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:44465 (local address 127.1.74.126:41225)
0926 06:57:21.781920 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.781922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.781988 (+ 66us) server_negotiation.cc:407] Connection header received
0926 06:57:21.782052 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.782052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.782059 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.782131 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:21.782514 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.783150 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.783971 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.784172 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.784283 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.785183 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:21.785184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:21.785184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:21.785190 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:21.786575 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.787000 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.787001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.787002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.787054 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.787300 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.787301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.787301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.787412 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:21.787485 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.787645 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.787717 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:57:21.787823 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.781793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41225 (local address 127.1.74.67:44465)
0926 06:57:21.781931 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.781942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.781999 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.782135 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.782135 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.782341 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:21.782423 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.782426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.783275 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.783276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.783877 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.783879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.784283 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.784284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.784363 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.784802 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:21.784807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:21.785079 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:21.786675 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.786676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.786676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.786918 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.787129 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.787130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.787130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.787208 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.787476 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:21.787477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:21.787530 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.787619 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.787725 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:21.806152 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.799549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.1.74.126:41225)
0926 06:57:21.799614 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.799616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.799624 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:21.799671 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.799671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.799680 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.799727 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:21.800279 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.800746 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.801497 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.801675 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.801727 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.803918 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:21.803920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:21.803920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:21.803926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:21.805264 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.805573 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.805574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.805574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.805623 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.805781 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.805781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.805781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.805885 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:21.805955 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.806039 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.806089 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:57:21.807928 3046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:21.809619 3046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:21.815970 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.811853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:34619 (local address 127.0.0.1:57342)
0926 06:57:21.812209 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.812246 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.812317 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.812737 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.812738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.812741 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:21.812836 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.812838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.813742 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.813743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.814754 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.814757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.815652 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.815653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.815736 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.815751 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.815838 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.815887 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":31,"thread_start_us":48,"threads_started":1}
I20260926 06:57:21.816217 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.811853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:44219 (local address 127.0.0.1:57762)
0926 06:57:21.812204 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.812245 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.812315 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.812716 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.812719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.812722 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:21.812804 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.812806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.813793 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.813794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.815050 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.815052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.815997 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.815998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.816075 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.816081 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.816117 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.816155 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":30,"thread_start_us":56,"threads_started":1}
I20260926 06:57:21.816015 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.812314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.1.74.66:34619)
0926 06:57:21.812469 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.812475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.812486 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:57:21.812527 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.812529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.812546 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.812671 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:21.812920 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.813653 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.814842 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.815558 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.815636 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.815703 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.815860 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.815900 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260926 06:57:21.816551 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.812039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.1.74.67:44219)
0926 06:57:21.812468 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.812477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.812487 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:21.812517 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.812519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.812532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.812916 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:21.812988 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.813697 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.815142 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.815897 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.816316 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.816387 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.816413 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.816451 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":158,"threads_started":1}
I20260926 06:57:21.817173 3324 tablet_service.cc:1467] Processing CreateTablet for tablet 81747e597bd543c1837d6536947b4b6b (DEFAULT_TABLE table=test-table [id=3f6cb58eda6c43c4b613cfe32f67562c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:21.817175 3464 tablet_service.cc:1467] Processing CreateTablet for tablet 81747e597bd543c1837d6536947b4b6b (DEFAULT_TABLE table=test-table [id=3f6cb58eda6c43c4b613cfe32f67562c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:21.817517 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.812068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40885 (local address 127.0.0.1:38326)
0926 06:57:21.812489 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.812500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.812544 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.814045 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.814045 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.814048 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:21.814081 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.814083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.815284 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.815285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.816251 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.816254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.817277 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.817278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.817336 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.817341 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.817396 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.817436 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":69,"threads_started":1}
I20260926 06:57:21.817548 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81747e597bd543c1837d6536947b4b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:21.817557 3464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81747e597bd543c1837d6536947b4b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:21.817780 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.812256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.1.74.65:40885)
0926 06:57:21.813878 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.813882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.813892 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:21.813933 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.813935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.813951 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.814036 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:21.814146 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.815168 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.816367 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.817104 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.817184 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.817238 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.817644 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.817684 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1583,"thread_start_us":85,"threads_started":1}
I20260926 06:57:21.820099 3184 tablet_service.cc:1467] Processing CreateTablet for tablet 81747e597bd543c1837d6536947b4b6b (DEFAULT_TABLE table=test-table [id=3f6cb58eda6c43c4b613cfe32f67562c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:21.820416 3184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81747e597bd543c1837d6536947b4b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:21.820432 3550 tablet_bootstrap.cc:492] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: Bootstrap starting.
I20260926 06:57:21.820780 3551 tablet_bootstrap.cc:492] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: Bootstrap starting.
I20260926 06:57:21.821440 3551 tablet_bootstrap.cc:654] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:21.821440 3550 tablet_bootstrap.cc:654] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:21.821784 3550 log.cc:824] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:21.821784 3551 log.cc:824] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:21.822597 3550 tablet_bootstrap.cc:492] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: No bootstrap required, opened a new log
I20260926 06:57:21.822655 3550 ts_tablet_manager.cc:1397] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:21.822765 3551 tablet_bootstrap.cc:492] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: No bootstrap required, opened a new log
I20260926 06:57:21.822813 3551 ts_tablet_manager.cc:1397] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:21.823319 3554 tablet_bootstrap.cc:492] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: Bootstrap starting.
I20260926 06:57:21.823917 3554 tablet_bootstrap.cc:654] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:21.824221 3554 log.cc:824] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:21.824930 3554 tablet_bootstrap.cc:492] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: No bootstrap required, opened a new log
I20260926 06:57:21.825030 3554 ts_tablet_manager.cc:1397] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:21.824932 3551 raft_consensus.cc:348] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.825109 3551 raft_consensus.cc:374] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:21.825022 3550 raft_consensus.cc:348] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.825155 3550 raft_consensus.cc:374] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:21.825150 3551 raft_consensus.cc:729] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b751efd090145f982765e75c0980b30, State: Initialized, Role: FOLLOWER
I20260926 06:57:21.825193 3550 raft_consensus.cc:729] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899c8b5a59741e9a704dd923f5f07b8, State: Initialized, Role: FOLLOWER
I20260926 06:57:21.825291 3551 consensus_queue.cc:260] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.825290 3550 consensus_queue.cc:260] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.825546 3551 ts_tablet_manager.cc:1428] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:57:21.825546 3550 ts_tablet_manager.cc:1428] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:57:21.825660 3530 heartbeater.cc:503] Master 127.1.74.126:41225 was elected leader, sending a full tablet report...
I20260926 06:57:21.825666 3390 heartbeater.cc:503] Master 127.1.74.126:41225 was elected leader, sending a full tablet report...
I20260926 06:57:21.826833 3554 raft_consensus.cc:348] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.826973 3554 raft_consensus.cc:374] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:21.827019 3554 raft_consensus.cc:729] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aefd9600b7840379c52972e0f92157c, State: Initialized, Role: FOLLOWER
I20260926 06:57:21.827142 3554 consensus_queue.cc:260] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.827411 3554 ts_tablet_manager.cc:1428] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:57:21.827518 3250 heartbeater.cc:503] Master 127.1.74.126:41225 was elected leader, sending a full tablet report...
W20260926 06:57:21.871824 3391 tablet.cc:2369] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:21.890880 3556 raft_consensus.cc:482] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:21.891001 3556 raft_consensus.cc:504] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.891341 3556 leader_election.cc:288] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619), 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885)
I20260926 06:57:21.894596 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.891553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:34619 (local address 127.1.74.67:41717)
0926 06:57:21.891614 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.891627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.891672 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.891922 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.891923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.891924 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:21.891979 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.891980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.892731 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.892732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.893679 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.893681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.894329 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.894330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.894420 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.894423 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.894490 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.894527 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 06:57:21.894596 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.891702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:41717 (local address 127.1.74.66:34619)
0926 06:57:21.891758 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.891761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.891770 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:21.891799 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.891800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.891812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.891873 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:21.892058 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.892653 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.893760 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.894252 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.894332 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.894343 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.894494 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.894527 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:57:21.894735 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.891790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:53047 (local address 127.1.74.65:40885)
0926 06:57:21.891848 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.891851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.891968 (+ 117us) server_negotiation.cc:407] Connection header received
0926 06:57:21.892063 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.892065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.892077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.892128 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:21.892348 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.892848 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.893808 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.894375 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.894453 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.894463 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.894612 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.894647 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:57:21.894935 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.891689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40885 (local address 127.1.74.67:53047)
0926 06:57:21.891887 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.891899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.891984 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.892195 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.892196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.892198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:21.892273 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.892275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.892928 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.892928 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.893707 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.893709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.894451 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.894453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.894535 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.894539 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.894609 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.894786 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":119,"threads_started":1}
I20260926 06:57:21.895160 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81747e597bd543c1837d6536947b4b6b" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b751efd090145f982765e75c0980b30" is_pre_election: true
I20260926 06:57:21.895242 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81747e597bd543c1837d6536947b4b6b" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aefd9600b7840379c52972e0f92157c" is_pre_election: true
I20260926 06:57:21.895349 3344 raft_consensus.cc:2454] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 0.
I20260926 06:57:21.895380 3204 raft_consensus.cc:2454] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 0.
I20260926 06:57:21.895583 3418 leader_election.cc:302] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b751efd090145f982765e75c0980b30, 9899c8b5a59741e9a704dd923f5f07b8; no voters:
I20260926 06:57:21.895725 3556 raft_consensus.cc:2781] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:21.895810 3556 raft_consensus.cc:482] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:21.895844 3556 raft_consensus.cc:3037] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:21.896682 3556 raft_consensus.cc:504] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.896821 3556 leader_election.cc:288] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 election: Requested vote from peers 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619), 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885)
I20260926 06:57:21.896988 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81747e597bd543c1837d6536947b4b6b" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b751efd090145f982765e75c0980b30"
I20260926 06:57:21.897007 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81747e597bd543c1837d6536947b4b6b" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aefd9600b7840379c52972e0f92157c"
I20260926 06:57:21.897073 3344 raft_consensus.cc:3037] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:21.897080 3204 raft_consensus.cc:3037] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:21.897986 3204 raft_consensus.cc:2454] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 1.
I20260926 06:57:21.897986 3344 raft_consensus.cc:2454] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 1.
I20260926 06:57:21.898156 3418 leader_election.cc:302] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aefd9600b7840379c52972e0f92157c, 9899c8b5a59741e9a704dd923f5f07b8; no voters:
I20260926 06:57:21.898265 3556 raft_consensus.cc:2781] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:21.898468 3556 raft_consensus.cc:686] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 LEADER]: Becoming Leader. State: Replica: 9899c8b5a59741e9a704dd923f5f07b8, State: Running, Role: LEADER
I20260926 06:57:21.898563 3556 consensus_queue.cc:237] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.899596 3045 catalog_manager.cc:5495] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "9899c8b5a59741e9a704dd923f5f07b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } health_report { overall_health: HEALTHY } } }
Sep 26 06:57:21 dist-test-slave-sfvt krb5kdc[3002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:21.927060 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.920004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41225 (local address 127.0.0.1:44300)
0926 06:57:21.920229 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.920240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.920315 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.920496 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.920497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.920804 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:21.920853 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.920855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.921480 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.921481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.921999 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.922000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.922306 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.922307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:21.922352 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.922830 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:21.922835 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:21.924470 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:21.926008 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.926015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.926017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.926287 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.926470 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:21.926471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:21.926472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:21.926542 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:21.926756 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:21.926761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:21.926805 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 06:57:21.926866 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:57:21.926959 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260926 06:57:21.927104 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.920124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.1.74.126:41225)
0926 06:57:21.920207 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.920209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.920308 (+ 99us) server_negotiation.cc:407] Connection header received
0926 06:57:21.920379 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.920380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.920388 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.920442 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:21.920925 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.921409 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.922070 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.922237 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.922284 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:21.924553 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:21.924555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:21.924555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:21.924561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:21.925928 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.926363 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.926364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.926364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.926411 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:21.926595 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:21.926596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:21.926596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:21.926688 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:21.926736 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:21.926898 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 06:57:21.927004 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":60}
I20260926 06:57:21.929189 3045 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:44300:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 06:57:21.929380 3045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:21.930786 3464 tablet_service.cc:1467] Processing CreateTablet for tablet a457b45b03bd437a876deeb51641463c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce9e3cbeb6994606ae771d0d3a1d1384]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:57:21.930781 3184 tablet_service.cc:1467] Processing CreateTablet for tablet a457b45b03bd437a876deeb51641463c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce9e3cbeb6994606ae771d0d3a1d1384]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:57:21.930819 3324 tablet_service.cc:1467] Processing CreateTablet for tablet a457b45b03bd437a876deeb51641463c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce9e3cbeb6994606ae771d0d3a1d1384]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:57:21.930961 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a457b45b03bd437a876deeb51641463c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:21.930961 3184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a457b45b03bd437a876deeb51641463c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:21.930961 3464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a457b45b03bd437a876deeb51641463c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:21.932514 3554 tablet_bootstrap.cc:492] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: Bootstrap starting.
I20260926 06:57:21.932574 3550 tablet_bootstrap.cc:492] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: Bootstrap starting.
I20260926 06:57:21.933036 3550 tablet_bootstrap.cc:654] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:21.933235 3554 tablet_bootstrap.cc:654] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:21.933574 3551 tablet_bootstrap.cc:492] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: Bootstrap starting.
I20260926 06:57:21.933786 3550 tablet_bootstrap.cc:492] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: No bootstrap required, opened a new log
I20260926 06:57:21.933832 3554 tablet_bootstrap.cc:492] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: No bootstrap required, opened a new log
I20260926 06:57:21.933830 3550 ts_tablet_manager.cc:1397] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:21.933869 3554 ts_tablet_manager.cc:1397] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:21.934010 3554 raft_consensus.cc:348] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.934010 3550 raft_consensus.cc:348] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.934073 3554 raft_consensus.cc:374] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:21.934073 3550 raft_consensus.cc:374] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:21.934095 3554 raft_consensus.cc:729] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aefd9600b7840379c52972e0f92157c, State: Initialized, Role: FOLLOWER
I20260926 06:57:21.934095 3550 raft_consensus.cc:729] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899c8b5a59741e9a704dd923f5f07b8, State: Initialized, Role: FOLLOWER
I20260926 06:57:21.934124 3551 tablet_bootstrap.cc:654] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:21.934145 3554 consensus_queue.cc:260] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.934145 3550 consensus_queue.cc:260] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.934230 3550 ts_tablet_manager.cc:1428] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.934230 3554 ts_tablet_manager.cc:1428] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:21.934320 3558 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:21.934326 3556 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:21.934425 3558 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:21.934428 3556 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:21.934964 3551 tablet_bootstrap.cc:492] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: No bootstrap required, opened a new log
I20260926 06:57:21.935012 3551 ts_tablet_manager.cc:1397] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:21.935148 3551 raft_consensus.cc:348] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.935213 3551 raft_consensus.cc:374] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:21.935227 3551 raft_consensus.cc:729] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b751efd090145f982765e75c0980b30, State: Initialized, Role: FOLLOWER
I20260926 06:57:21.935307 3551 consensus_queue.cc:260] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.935389 3551 ts_tablet_manager.cc:1428] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:21.935451 3557 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:21.935555 3557 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 06:57:21.964957 3251 tablet.cc:2369] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:21.996630 3558 raft_consensus.cc:482] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:21.996729 3558 raft_consensus.cc:504] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:21.997110 3558 leader_election.cc:288] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619), 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67:44219)
I20260926 06:57:22.001081 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.997373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:39597 (local address 127.1.74.67:44219)
0926 06:57:21.997417 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.997420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.997498 (+ 78us) server_negotiation.cc:407] Connection header received
0926 06:57:21.997560 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.997561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.997572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.997635 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:21.997799 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.998534 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.000168 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.000719 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.000796 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.000807 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:22.000971 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 06:57:22.001009 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:57:22.001081 3549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.997298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:41531 (local address 127.1.74.66:34619)
0926 06:57:21.997353 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:21.997356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:21.997499 (+ 143us) server_negotiation.cc:407] Connection header received
0926 06:57:21.997560 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:21.997561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:21.997571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:21.997635 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:21.997800 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.998577 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.000167 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.000719 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.000796 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.000807 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:22.000973 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 06:57:22.001009 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:57:22.001133 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.997255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:44219 (local address 127.1.74.65:39597)
0926 06:57:21.997414 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.997426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.997501 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.997652 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.997652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.997654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:21.997725 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.997727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.998603 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.998604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.000078 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:22.000081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.000811 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.000813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.000888 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.000892 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:22.000953 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:57:22.001009 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":70,"thread_start_us":72,"threads_started":1}
I20260926 06:57:22.001412 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:21.997256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:34619 (local address 127.1.74.65:41531)
0926 06:57:21.997440 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:21.997450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:21.997501 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:21.997652 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:21.997652 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:21.997671 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:21.997726 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:21.997727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:21.998640 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:21.998640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.000078 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:22.000081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.000813 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.000813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.000893 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.000895 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:57:22.000953 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 06:57:22.001009 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":65,"thread_start_us":86,"threads_started":1}
I20260926 06:57:22.001605 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a457b45b03bd437a876deeb51641463c" candidate_uuid: "5aefd9600b7840379c52972e0f92157c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899c8b5a59741e9a704dd923f5f07b8" is_pre_election: true
I20260926 06:57:22.001749 3484 raft_consensus.cc:2454] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aefd9600b7840379c52972e0f92157c in term 0.
I20260926 06:57:22.001724 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a457b45b03bd437a876deeb51641463c" candidate_uuid: "5aefd9600b7840379c52972e0f92157c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b751efd090145f982765e75c0980b30" is_pre_election: true
I20260926 06:57:22.001835 3344 raft_consensus.cc:2454] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aefd9600b7840379c52972e0f92157c in term 0.
I20260926 06:57:22.001953 3135 leader_election.cc:302] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aefd9600b7840379c52972e0f92157c, 9899c8b5a59741e9a704dd923f5f07b8; no voters:
I20260926 06:57:22.002106 3558 raft_consensus.cc:2781] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:22.002164 3558 raft_consensus.cc:482] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:22.002195 3558 raft_consensus.cc:3037] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:22.002892 3558 raft_consensus.cc:504] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.003046 3558 leader_election.cc:288] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [CANDIDATE]: Term 1 election: Requested vote from peers 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619), 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67:44219)
I20260926 06:57:22.003211 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a457b45b03bd437a876deeb51641463c" candidate_uuid: "5aefd9600b7840379c52972e0f92157c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b751efd090145f982765e75c0980b30"
I20260926 06:57:22.003343 3344 raft_consensus.cc:3037] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:22.003355 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a457b45b03bd437a876deeb51641463c" candidate_uuid: "5aefd9600b7840379c52972e0f92157c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899c8b5a59741e9a704dd923f5f07b8"
I20260926 06:57:22.003448 3484 raft_consensus.cc:3037] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:22.004004 3344 raft_consensus.cc:2454] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aefd9600b7840379c52972e0f92157c in term 1.
I20260926 06:57:22.004074 3484 raft_consensus.cc:2454] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aefd9600b7840379c52972e0f92157c in term 1.
I20260926 06:57:22.004204 3138 leader_election.cc:302] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aefd9600b7840379c52972e0f92157c, 7b751efd090145f982765e75c0980b30; no voters:
I20260926 06:57:22.004293 3558 raft_consensus.cc:2781] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:22.004475 3558 raft_consensus.cc:686] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [term 1 LEADER]: Becoming Leader. State: Replica: 5aefd9600b7840379c52972e0f92157c, State: Running, Role: LEADER
I20260926 06:57:22.004575 3558 consensus_queue.cc:237] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.004998 3573 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: TxnStatusTablet state changed. Reason: New leader 5aefd9600b7840379c52972e0f92157c. Latest consensus state: current_term: 1 leader_uuid: "5aefd9600b7840379c52972e0f92157c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:22.005079 3573 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:57:22.005293 3575 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 06:57:22.005409 3575 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 06:57:22.005369 3045 catalog_manager.cc:5495] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aefd9600b7840379c52972e0f92157c (127.1.74.65). New cstate: current_term: 1 leader_uuid: "5aefd9600b7840379c52972e0f92157c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } health_report { overall_health: UNKNOWN } } }
I20260926 06:57:22.028187 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:22.025717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.1.74.65:40885)
0926 06:57:22.025763 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:22.025765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:22.025775 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:22.025867 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:22.025868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:22.025886 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:22.025951 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:22.026092 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.026587 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.027355 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.027573 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.027646 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.027828 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:22.027938 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:22.028017 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:22.028070 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 06:57:22.028091 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:57:22.028190 3567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:22.025633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40885 (local address 127.0.0.1:38340)
0926 06:57:22.025712 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:22.025724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:22.025773 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:22.025962 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:22.025963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:22.025964 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:22.026014 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:22.026016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.026676 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.026677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.027245 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:22.027246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.027644 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.027645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.027707 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.027724 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:22.028008 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:22.028015 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 06:57:22.028056 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 06:57:22.028104 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 06:57:22.028970 3531 tablet.cc:2369] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:22.030014 3484 raft_consensus.cc:1261] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Refusing update from remote peer 5aefd9600b7840379c52972e0f92157c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:22.030082 3344 raft_consensus.cc:1261] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Refusing update from remote peer 5aefd9600b7840379c52972e0f92157c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:22.030308 3573 consensus_queue.cc:1035] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:22.030483 3558 consensus_queue.cc:1035] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:22.031484 3556 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: TxnStatusTablet state changed. Reason: New leader 5aefd9600b7840379c52972e0f92157c. Latest consensus state: current_term: 1 leader_uuid: "5aefd9600b7840379c52972e0f92157c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:22.031584 3556 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:22.031688 3557 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: TxnStatusTablet state changed. Reason: New leader 5aefd9600b7840379c52972e0f92157c. Latest consensus state: current_term: 1 leader_uuid: "5aefd9600b7840379c52972e0f92157c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:22.031783 3557 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:22.032465 3578 mvcc.cc:205] Tried to move back new op lower bound from 7333502328952537088 to 7333502328851341312. Current Snapshot: MvccSnapshot[applied={T|T < 7333502328952537088}]
I20260926 06:57:22.032559 3573 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aefd9600b7840379c52972e0f92157c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:22.032655 3573 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:57:22.032698 3573 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aefd9600b7840379c52972e0f92157c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:22.032737 3573 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 5aefd9600b7840379c52972e0f92157c: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:57:22.033493 3556 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aefd9600b7840379c52972e0f92157c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:22.033569 3556 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 9899c8b5a59741e9a704dd923f5f07b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:22.034759 3557 tablet_replica.cc:439] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aefd9600b7840379c52972e0f92157c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } } }
I20260926 06:57:22.034879 3557 tablet_replica.cc:442] T a457b45b03bd437a876deeb51641463c P 7b751efd090145f982765e75c0980b30: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:22.037607 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:22.035264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.1.74.67:44219)
0926 06:57:22.035341 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:22.035343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:22.035351 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:22.035393 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:22.035394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:22.035409 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:22.035472 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:22.035573 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.036044 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.036820 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.037030 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.037114 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.037290 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:22.037398 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:22.037488 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:22.037511 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 06:57:22.037532 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 06:57:22.042339 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:22.039804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40885 (local address 127.1.74.67:46905)
0926 06:57:22.039860 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:22.039871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:22.039913 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:22.040053 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:22.040054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:22.040055 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:22.040099 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:22.040101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.040962 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.040962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.041525 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:22.041527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.041986 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.041987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.042038 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.042049 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:22.042223 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:22.042224 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:57:22.042259 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 06:57:22.042292 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 06:57:22.042351 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:22.039868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:46905 (local address 127.1.74.65:40885)
0926 06:57:22.039908 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:22.039910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:22.039919 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:22.039975 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:22.039976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:22.039988 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:22.040039 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:22.040165 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.040882 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.041612 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.041917 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.041984 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.042131 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:22.042165 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:22.042217 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:22.042262 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 06:57:22.042279 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:57:22.044535 3344 raft_consensus.cc:1261] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Refusing update from remote peer 9899c8b5a59741e9a704dd923f5f07b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:22.044559 3204 raft_consensus.cc:1261] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Refusing update from remote peer 9899c8b5a59741e9a704dd923f5f07b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:22.044760 3556 consensus_queue.cc:1035] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:22.044875 3560 consensus_queue.cc:1035] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:22.054006 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:22.051590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:42521 (local address 127.1.74.67:44219)
0926 06:57:22.051656 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:22.051658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:22.051667 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:22.051713 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:22.051714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:22.051725 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:22.051778 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:22.051917 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.052404 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.053193 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.053466 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.053536 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.053720 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:22.053758 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:22.053814 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:22.053910 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 06:57:22.053931 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:57:22.054075 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:22.051435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:44219 (local address 127.1.74.65:42521)
0926 06:57:22.051578 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:22.051589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:22.051663 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:22.051783 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:22.051784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:22.051785 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:22.051855 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:22.051865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.052488 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.052489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.053097 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:22.053106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:22.053546 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:22.053547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:22.053616 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:22.053633 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:22.053826 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:22.053828 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:57:22.053908 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:57:22.053954 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260926 06:57:22.064180 3047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:22.064294 3047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:22.064975 3047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: Sending DeleteTablet for 3 replicas of tablet 81747e597bd543c1837d6536947b4b6b
I20260926 06:57:22.065311 3324 tablet_service.cc:1514] Processing DeleteTablet for tablet 81747e597bd543c1837d6536947b4b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:22 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57342
I20260926 06:57:22.065361 3184 tablet_service.cc:1514] Processing DeleteTablet for tablet 81747e597bd543c1837d6536947b4b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:22 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:38326
I20260926 06:57:22.065594 3464 tablet_service.cc:1514] Processing DeleteTablet for tablet 81747e597bd543c1837d6536947b4b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:22 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57762
I20260926 06:57:22.065652 3598 tablet_replica.cc:330] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: stopping tablet replica
I20260926 06:57:22.065837 3598 raft_consensus.cc:2229] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:22.065951 3598 raft_consensus.cc:2258] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:22.066047 3047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:22.066188 3047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:22.066642 3598 ts_tablet_manager.cc:1905] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:22.067276 3324 tablet_service.cc:1467] Processing CreateTablet for tablet 0b36a1a5a277458d91301175bcb303e5 (DEFAULT_TABLE table=test-table [id=3a47de533ffd45499e99a535d3bc53db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:22.067399 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b36a1a5a277458d91301175bcb303e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:22.067389 3184 tablet_service.cc:1467] Processing CreateTablet for tablet 0b36a1a5a277458d91301175bcb303e5 (DEFAULT_TABLE table=test-table [id=3a47de533ffd45499e99a535d3bc53db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:22.067495 3184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b36a1a5a277458d91301175bcb303e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:22.068706 3551 tablet_bootstrap.cc:492] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: Bootstrap starting.
I20260926 06:57:22.068708 3554 tablet_bootstrap.cc:492] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: Bootstrap starting.
I20260926 06:57:22.069120 3554 tablet_bootstrap.cc:654] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:22.069120 3551 tablet_bootstrap.cc:654] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:22.069619 3464 tablet_service.cc:1467] Processing CreateTablet for tablet 0b36a1a5a277458d91301175bcb303e5 (DEFAULT_TABLE table=test-table [id=3a47de533ffd45499e99a535d3bc53db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:22.069748 3464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b36a1a5a277458d91301175bcb303e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:22.069852 3597 tablet_replica.cc:330] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: stopping tablet replica
I20260926 06:57:22.070080 3597 raft_consensus.cc:2229] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:22.070875 3598 ts_tablet_manager.cc:1918] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:22.071012 3550 tablet_bootstrap.cc:492] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: Bootstrap starting.
I20260926 06:57:22.071013 3597 raft_consensus.cc:2258] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:22.071228 3598 log.cc:1196] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/81747e597bd543c1837d6536947b4b6b
I20260926 06:57:22.071455 3550 tablet_bootstrap.cc:654] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:22.071610 3597 ts_tablet_manager.cc:1905] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:22.071655 3598 ts_tablet_manager.cc:1939] T 81747e597bd543c1837d6536947b4b6b P 5aefd9600b7840379c52972e0f92157c: Deleting consensus metadata
I20260926 06:57:22.072147 3034 catalog_manager.cc:4841] TS 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885): tablet 81747e597bd543c1837d6536947b4b6b (table test-table [id=3f6cb58eda6c43c4b613cfe32f67562c]) successfully deleted
I20260926 06:57:22.072359 3599 tablet_replica.cc:330] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: stopping tablet replica
I20260926 06:57:22.072512 3599 raft_consensus.cc:2229] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:22.072651 3550 tablet_bootstrap.cc:492] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: No bootstrap required, opened a new log
I20260926 06:57:22.072702 3550 ts_tablet_manager.cc:1397] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:22.072849 3554 tablet_bootstrap.cc:492] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: No bootstrap required, opened a new log
I20260926 06:57:22.072883 3554 ts_tablet_manager.cc:1397] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 06:57:22.072904 3550 raft_consensus.cc:348] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.072983 3550 raft_consensus.cc:374] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:22.073004 3550 raft_consensus.cc:729] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899c8b5a59741e9a704dd923f5f07b8, State: Initialized, Role: FOLLOWER
I20260926 06:57:22.073045 3550 consensus_queue.cc:260] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.073140 3550 ts_tablet_manager.cc:1428] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:22.073135 3554 raft_consensus.cc:348] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.073223 3554 raft_consensus.cc:374] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:22.073244 3554 raft_consensus.cc:729] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aefd9600b7840379c52972e0f92157c, State: Initialized, Role: FOLLOWER
I20260926 06:57:22.073282 3554 consensus_queue.cc:260] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.073361 3554 ts_tablet_manager.cc:1428] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:22.072681 3599 raft_consensus.cc:2258] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:22.074043 3599 ts_tablet_manager.cc:1905] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:22.075659 3551 tablet_bootstrap.cc:492] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: No bootstrap required, opened a new log
I20260926 06:57:22.075709 3551 ts_tablet_manager.cc:1397] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260926 06:57:22.075770 3599 ts_tablet_manager.cc:1918] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:22.075888 3599 log.cc:1196] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/81747e597bd543c1837d6536947b4b6b
I20260926 06:57:22.075860 3551 raft_consensus.cc:348] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.075923 3551 raft_consensus.cc:374] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:22.075948 3551 raft_consensus.cc:729] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b751efd090145f982765e75c0980b30, State: Initialized, Role: FOLLOWER
I20260926 06:57:22.075984 3551 consensus_queue.cc:260] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.076086 3551 ts_tablet_manager.cc:1428] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:22.076323 3599 ts_tablet_manager.cc:1939] T 81747e597bd543c1837d6536947b4b6b P 9899c8b5a59741e9a704dd923f5f07b8: Deleting consensus metadata
I20260926 06:57:22.076740 3031 catalog_manager.cc:4841] TS 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67:44219): tablet 81747e597bd543c1837d6536947b4b6b (table test-table [id=3f6cb58eda6c43c4b613cfe32f67562c]) successfully deleted
I20260926 06:57:22.083415 3597 ts_tablet_manager.cc:1918] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:22.083514 3597 log.cc:1196] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/81747e597bd543c1837d6536947b4b6b
I20260926 06:57:22.083756 3597 ts_tablet_manager.cc:1939] T 81747e597bd543c1837d6536947b4b6b P 7b751efd090145f982765e75c0980b30: Deleting consensus metadata
I20260926 06:57:22.084228 3034 catalog_manager.cc:4841] TS 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619): tablet 81747e597bd543c1837d6536947b4b6b (table test-table [id=3f6cb58eda6c43c4b613cfe32f67562c]) successfully deleted
I20260926 06:57:22.111316 3560 raft_consensus.cc:482] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:22.111421 3560 raft_consensus.cc:504] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.111593 3560 leader_election.cc:288] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885), 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619)
I20260926 06:57:22.111817 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b36a1a5a277458d91301175bcb303e5" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aefd9600b7840379c52972e0f92157c" is_pre_election: true
I20260926 06:57:22.111830 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b36a1a5a277458d91301175bcb303e5" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b751efd090145f982765e75c0980b30" is_pre_election: true
I20260926 06:57:22.111907 3204 raft_consensus.cc:2454] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 0.
I20260926 06:57:22.111945 3344 raft_consensus.cc:2454] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 0.
I20260926 06:57:22.112058 3418 leader_election.cc:302] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aefd9600b7840379c52972e0f92157c, 9899c8b5a59741e9a704dd923f5f07b8; no voters:
I20260926 06:57:22.112167 3560 raft_consensus.cc:2781] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:22.112200 3560 raft_consensus.cc:482] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:22.112221 3560 raft_consensus.cc:3037] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:22.113067 3560 raft_consensus.cc:504] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.113188 3560 leader_election.cc:288] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 election: Requested vote from peers 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885), 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619)
I20260926 06:57:22.113387 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b36a1a5a277458d91301175bcb303e5" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aefd9600b7840379c52972e0f92157c"
I20260926 06:57:22.113402 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b36a1a5a277458d91301175bcb303e5" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b751efd090145f982765e75c0980b30"
I20260926 06:57:22.113466 3204 raft_consensus.cc:3037] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:22.113468 3344 raft_consensus.cc:3037] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:22.114187 3344 raft_consensus.cc:2454] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 1.
I20260926 06:57:22.114262 3204 raft_consensus.cc:2454] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 1.
I20260926 06:57:22.114341 3418 leader_election.cc:302] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b751efd090145f982765e75c0980b30, 9899c8b5a59741e9a704dd923f5f07b8; no voters:
I20260926 06:57:22.114444 3560 raft_consensus.cc:2781] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:22.114495 3560 raft_consensus.cc:686] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 LEADER]: Becoming Leader. State: Replica: 9899c8b5a59741e9a704dd923f5f07b8, State: Running, Role: LEADER
I20260926 06:57:22.114545 3560 consensus_queue.cc:237] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:22.115087 3045 catalog_manager.cc:5495] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "9899c8b5a59741e9a704dd923f5f07b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } health_report { overall_health: HEALTHY } } }
I20260926 06:57:22.125020 3204 raft_consensus.cc:1261] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Refusing update from remote peer 9899c8b5a59741e9a704dd923f5f07b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:22.125299 3344 raft_consensus.cc:1261] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Refusing update from remote peer 9899c8b5a59741e9a704dd923f5f07b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:22.125267 3556 consensus_queue.cc:1035] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:22.125495 3556 consensus_queue.cc:1035] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:22.127159 3580 mvcc.cc:205] Tried to move back new op lower bound from 7333502329342541824 to 7333502329301295104. Current Snapshot: MvccSnapshot[applied={T|T < 7333502329342541824}]
W20260926 06:57:22.131850 3579 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 06:57:23.652683 3045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:23.652784 3045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:23.653599 3045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: Sending DeleteTablet for 3 replicas of tablet 0b36a1a5a277458d91301175bcb303e5
I20260926 06:57:23.653903 3464 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b36a1a5a277458d91301175bcb303e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:23 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57762
I20260926 06:57:23.653903 3324 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b36a1a5a277458d91301175bcb303e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:23 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57342
I20260926 06:57:23.654088 3184 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b36a1a5a277458d91301175bcb303e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:23 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:38326
I20260926 06:57:23.654258 3631 tablet_replica.cc:330] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: stopping tablet replica
I20260926 06:57:23.654266 3629 tablet_replica.cc:330] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: stopping tablet replica
I20260926 06:57:23.654310 3631 raft_consensus.cc:2229] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:23.654310 3629 raft_consensus.cc:2229] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:23.654345 3631 raft_consensus.cc:2258] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:23.654364 3629 raft_consensus.cc:2258] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:23.654476 3630 tablet_replica.cc:330] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: stopping tablet replica
I20260926 06:57:23.654526 3630 raft_consensus.cc:2229] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:23.654573 3630 raft_consensus.cc:2258] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:23.654641 3631 ts_tablet_manager.cc:1905] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:23.654754 3629 ts_tablet_manager.cc:1905] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:23.656576 3630 ts_tablet_manager.cc:1905] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:23.656711 3629 ts_tablet_manager.cc:1918] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:57:23.656729 3631 ts_tablet_manager.cc:1918] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:57:23.656778 3629 log.cc:1196] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/0b36a1a5a277458d91301175bcb303e5
I20260926 06:57:23.656783 3631 log.cc:1196] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/0b36a1a5a277458d91301175bcb303e5
I20260926 06:57:23.657027 3631 ts_tablet_manager.cc:1939] T 0b36a1a5a277458d91301175bcb303e5 P 5aefd9600b7840379c52972e0f92157c: Deleting consensus metadata
I20260926 06:57:23.657051 3629 ts_tablet_manager.cc:1939] T 0b36a1a5a277458d91301175bcb303e5 P 9899c8b5a59741e9a704dd923f5f07b8: Deleting consensus metadata
I20260926 06:57:23.657455 3031 catalog_manager.cc:4841] TS 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67:44219): tablet 0b36a1a5a277458d91301175bcb303e5 (table test-table [id=3a47de533ffd45499e99a535d3bc53db]) successfully deleted
I20260926 06:57:23.657471 3034 catalog_manager.cc:4841] TS 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885): tablet 0b36a1a5a277458d91301175bcb303e5 (table test-table [id=3a47de533ffd45499e99a535d3bc53db]) successfully deleted
I20260926 06:57:23.660450 3630 ts_tablet_manager.cc:1918] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:57:23.660523 3630 log.cc:1196] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/0b36a1a5a277458d91301175bcb303e5
I20260926 06:57:23.660753 3630 ts_tablet_manager.cc:1939] T 0b36a1a5a277458d91301175bcb303e5 P 7b751efd090145f982765e75c0980b30: Deleting consensus metadata
I20260926 06:57:23.661077 3034 catalog_manager.cc:4841] TS 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619): tablet 0b36a1a5a277458d91301175bcb303e5 (table test-table [id=3a47de533ffd45499e99a535d3bc53db]) successfully deleted
Sep 26 06:57:23 dist-test-slave-sfvt krb5kdc[3002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405843, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:57:23 dist-test-slave-sfvt krb5kdc[3002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405843, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:23.677683 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.670544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.1.74.126:41225)
0926 06:57:23.670692 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:23.670695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:23.670731 (+ 36us) server_negotiation.cc:407] Connection header received
0926 06:57:23.670852 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:23.670853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:23.670865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:23.670946 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:23.671586 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.672092 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.672837 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.673031 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.673112 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.675370 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:23.675371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:23.675371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:23.675378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:23.676736 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:23.677071 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:23.677072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:23.677072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:23.677122 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:23.677282 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:23.677282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:23.677282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:23.677381 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:23.677453 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:23.677551 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 06:57:23.677614 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260926 06:57:23.679203 3045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:23.679411 3045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:23.680709 3324 tablet_service.cc:1467] Processing CreateTablet for tablet e11105ef9aea44668a5763f8977275a4 (DEFAULT_TABLE table=test-table [id=5887dce8104549c69727ca9aa4adacad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:23.680740 3184 tablet_service.cc:1467] Processing CreateTablet for tablet e11105ef9aea44668a5763f8977275a4 (DEFAULT_TABLE table=test-table [id=5887dce8104549c69727ca9aa4adacad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:23.680845 3184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11105ef9aea44668a5763f8977275a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:23.680846 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11105ef9aea44668a5763f8977275a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:23.682260 3641 tablet_bootstrap.cc:492] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: Bootstrap starting.
I20260926 06:57:23.682283 3464 tablet_service.cc:1467] Processing CreateTablet for tablet e11105ef9aea44668a5763f8977275a4 (DEFAULT_TABLE table=test-table [id=5887dce8104549c69727ca9aa4adacad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:23.682408 3464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e11105ef9aea44668a5763f8977275a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:23.682688 3641 tablet_bootstrap.cc:654] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:23.682984 3640 tablet_bootstrap.cc:492] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: Bootstrap starting.
I20260926 06:57:23.683311 3641 tablet_bootstrap.cc:492] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: No bootstrap required, opened a new log
I20260926 06:57:23.683363 3641 ts_tablet_manager.cc:1397] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:23.683437 3640 tablet_bootstrap.cc:654] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:23.683485 3641 raft_consensus.cc:348] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.683566 3641 raft_consensus.cc:374] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:23.683593 3641 raft_consensus.cc:729] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aefd9600b7840379c52972e0f92157c, State: Initialized, Role: FOLLOWER
I20260926 06:57:23.683647 3641 consensus_queue.cc:260] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.683774 3641 ts_tablet_manager.cc:1428] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:23.683993 3643 tablet_bootstrap.cc:492] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: Bootstrap starting.
I20260926 06:57:23.684414 3643 tablet_bootstrap.cc:654] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:23.684974 3643 tablet_bootstrap.cc:492] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: No bootstrap required, opened a new log
I20260926 06:57:23.684994 3640 tablet_bootstrap.cc:492] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: No bootstrap required, opened a new log
I20260926 06:57:23.685029 3640 ts_tablet_manager.cc:1397] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:23.685042 3643 ts_tablet_manager.cc:1397] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:23.685163 3643 raft_consensus.cc:348] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.685170 3640 raft_consensus.cc:348] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.685240 3640 raft_consensus.cc:374] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:23.685241 3643 raft_consensus.cc:374] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:23.685266 3643 raft_consensus.cc:729] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899c8b5a59741e9a704dd923f5f07b8, State: Initialized, Role: FOLLOWER
I20260926 06:57:23.685252 3640 raft_consensus.cc:729] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b751efd090145f982765e75c0980b30, State: Initialized, Role: FOLLOWER
I20260926 06:57:23.685338 3643 consensus_queue.cc:260] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.685348 3640 consensus_queue.cc:260] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.685454 3643 ts_tablet_manager.cc:1428] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:23.685534 3640 ts_tablet_manager.cc:1428] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:23.743803 3646 raft_consensus.cc:482] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:23.743896 3646 raft_consensus.cc:504] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.744253 3646 leader_election.cc:288] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885), 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67:44219)
I20260926 06:57:23.748172 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.744501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:40821 (local address 127.1.74.65:40885)
0926 06:57:23.744664 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:23.744667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:23.744679 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:57:23.744766 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:23.744767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:23.744779 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:23.744866 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:23.745030 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.745773 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.747228 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.747808 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.747886 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.747908 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:23.748060 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 06:57:23.748098 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260926 06:57:23.748172 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.744500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:46921 (local address 127.1.74.67:44219)
0926 06:57:23.744664 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:23.744667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:23.744679 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:57:23.744766 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:23.744767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:23.744778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:23.744865 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:23.745362 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.746032 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.747224 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.747808 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.747886 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.747897 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:23.748050 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 06:57:23.748098 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260926 06:57:23.748202 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.744429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:44219 (local address 127.1.74.66:46921)
0926 06:57:23.744621 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:23.744634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:23.744692 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:23.744993 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:23.744994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:23.744996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:23.745286 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:23.745288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.746095 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.746096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:23.747137 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:23.747139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.747896 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.747897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:23.747966 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.747973 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 06:57:23.748034 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:57:23.748083 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"spinlock_wait_cycles":896,"thread_start_us":137,"threads_started":1}
I20260926 06:57:23.748205 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.744429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40885 (local address 127.1.74.66:40821)
0926 06:57:23.744627 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:23.744637 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:23.744692 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:23.744879 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:23.744881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:23.744883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:23.744960 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:23.744961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.745839 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.745840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:23.747133 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:23.747136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.747895 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.747896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:23.747966 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.747975 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:57:23.748034 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:57:23.748083 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":12,"thread_start_us":114,"threads_started":1}
I20260926 06:57:23.748528 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11105ef9aea44668a5763f8977275a4" candidate_uuid: "7b751efd090145f982765e75c0980b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899c8b5a59741e9a704dd923f5f07b8" is_pre_election: true
I20260926 06:57:23.748622 3484 raft_consensus.cc:2454] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b751efd090145f982765e75c0980b30 in term 0.
I20260926 06:57:23.748654 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11105ef9aea44668a5763f8977275a4" candidate_uuid: "7b751efd090145f982765e75c0980b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aefd9600b7840379c52972e0f92157c" is_pre_election: true
I20260926 06:57:23.748760 3204 raft_consensus.cc:2454] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b751efd090145f982765e75c0980b30 in term 0.
I20260926 06:57:23.748796 3275 leader_election.cc:302] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b751efd090145f982765e75c0980b30, 9899c8b5a59741e9a704dd923f5f07b8; no voters:
I20260926 06:57:23.748971 3646 raft_consensus.cc:2781] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:23.749013 3646 raft_consensus.cc:482] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:23.749029 3646 raft_consensus.cc:3037] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:23.749712 3646 raft_consensus.cc:504] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.749857 3646 leader_election.cc:288] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [CANDIDATE]: Term 1 election: Requested vote from peers 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885), 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67:44219)
I20260926 06:57:23.750000 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11105ef9aea44668a5763f8977275a4" candidate_uuid: "7b751efd090145f982765e75c0980b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899c8b5a59741e9a704dd923f5f07b8"
I20260926 06:57:23.750002 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e11105ef9aea44668a5763f8977275a4" candidate_uuid: "7b751efd090145f982765e75c0980b30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aefd9600b7840379c52972e0f92157c"
I20260926 06:57:23.750099 3484 raft_consensus.cc:3037] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:23.750098 3204 raft_consensus.cc:3037] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:23.750680 3484 raft_consensus.cc:2454] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b751efd090145f982765e75c0980b30 in term 1.
I20260926 06:57:23.750728 3204 raft_consensus.cc:2454] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b751efd090145f982765e75c0980b30 in term 1.
I20260926 06:57:23.750905 3278 leader_election.cc:302] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aefd9600b7840379c52972e0f92157c, 7b751efd090145f982765e75c0980b30; no voters:
I20260926 06:57:23.750993 3646 raft_consensus.cc:2781] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:23.751135 3646 raft_consensus.cc:686] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 1 LEADER]: Becoming Leader. State: Replica: 7b751efd090145f982765e75c0980b30, State: Running, Role: LEADER
I20260926 06:57:23.751215 3646 consensus_queue.cc:237] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.751950 3045 catalog_manager.cc:5495] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b751efd090145f982765e75c0980b30 (127.1.74.66). New cstate: current_term: 1 leader_uuid: "7b751efd090145f982765e75c0980b30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } health_report { overall_health: UNKNOWN } } }
I20260926 06:57:23.759907 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.757305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.1.74.66:34619)
0926 06:57:23.757477 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:23.757479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:23.757487 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:23.757526 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:23.757526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:23.757543 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:23.757634 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:23.757809 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.758287 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.759151 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.759408 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.759460 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.759597 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:23.759695 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:23.759745 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:23.759778 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 06:57:23.759792 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260926 06:57:23.764223 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.762179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:34603 (local address 127.1.74.65:40885)
0926 06:57:23.762244 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:23.762246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:23.762294 (+ 48us) server_negotiation.cc:407] Connection header received
0926 06:57:23.762335 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:23.762335 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:23.762345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:23.762390 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:23.762504 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.762930 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.763680 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.763856 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.763907 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.764037 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:23.764065 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:23.764101 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:23.764157 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 06:57:23.764173 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:57:23.764237 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.762110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40885 (local address 127.1.74.66:34603)
0926 06:57:23.762233 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:23.762243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:23.762298 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:23.762403 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:23.762403 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:23.762405 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:23.762454 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:23.762454 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.762985 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.762986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:23.763616 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:23.763618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.763916 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.763917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:23.763966 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.763976 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:23.764112 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:23.764114 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:57:23.764148 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 06:57:23.764181 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
I20260926 06:57:23.766137 3484 raft_consensus.cc:1261] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Refusing update from remote peer 7b751efd090145f982765e75c0980b30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:23.766181 3204 raft_consensus.cc:1261] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Refusing update from remote peer 7b751efd090145f982765e75c0980b30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:23.766412 3651 consensus_queue.cc:1035] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:23.766541 3646 consensus_queue.cc:1035] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:23.773510 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.771072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:35529 (local address 127.1.74.66:34619)
0926 06:57:23.771105 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:23.771107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:23.771212 (+ 105us) server_negotiation.cc:407] Connection header received
0926 06:57:23.771294 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:23.771295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:23.771304 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:23.771374 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:23.771471 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.771925 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.772782 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.773059 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.773130 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.773280 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:23.773309 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:23.773376 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:23.773427 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 06:57:23.773446 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:57:23.773546 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.771036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:34619 (local address 127.1.74.65:35529)
0926 06:57:23.771147 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:23.771157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:23.771226 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:23.771378 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:23.771378 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:23.771380 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:23.771422 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:23.771424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.772014 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.772015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:23.772709 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:23.772712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.773140 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.773141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:23.773195 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.773199 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:23.773379 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:23.773381 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:57:23.773426 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 06:57:23.773470 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":71,"threads_started":1}
I20260926 06:57:23.779702 3047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:23.779811 3047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:23.780591 3047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: Sending DeleteTablet for 3 replicas of tablet e11105ef9aea44668a5763f8977275a4
I20260926 06:57:23.780795 3184 tablet_service.cc:1514] Processing DeleteTablet for tablet e11105ef9aea44668a5763f8977275a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:23 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:38326
I20260926 06:57:23.780827 3464 tablet_service.cc:1514] Processing DeleteTablet for tablet e11105ef9aea44668a5763f8977275a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:23 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57762
I20260926 06:57:23.780913 3631 tablet_replica.cc:330] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: stopping tablet replica
I20260926 06:57:23.780925 3629 tablet_replica.cc:330] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: stopping tablet replica
I20260926 06:57:23.780944 3324 tablet_service.cc:1514] Processing DeleteTablet for tablet e11105ef9aea44668a5763f8977275a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:23 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57342
I20260926 06:57:23.780978 3629 raft_consensus.cc:2229] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:23.780982 3631 raft_consensus.cc:2229] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:23.781015 3629 raft_consensus.cc:2258] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:23.781018 3630 tablet_replica.cc:330] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: stopping tablet replica
I20260926 06:57:23.781028 3631 raft_consensus.cc:2258] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:23.781075 3630 raft_consensus.cc:2229] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:23.781158 3630 raft_consensus.cc:2258] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:23.781419 3631 ts_tablet_manager.cc:1905] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:23.781467 3629 ts_tablet_manager.cc:1905] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:23.783164 3630 ts_tablet_manager.cc:1905] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:23.783162 3631 ts_tablet_manager.cc:1918] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:23.783218 3631 log.cc:1196] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/e11105ef9aea44668a5763f8977275a4
I20260926 06:57:23.783464 3631 ts_tablet_manager.cc:1939] T e11105ef9aea44668a5763f8977275a4 P 5aefd9600b7840379c52972e0f92157c: Deleting consensus metadata
I20260926 06:57:23.783921 3034 catalog_manager.cc:4841] TS 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885): tablet e11105ef9aea44668a5763f8977275a4 (table test-table [id=5887dce8104549c69727ca9aa4adacad]) successfully deleted
I20260926 06:57:23.784430 3630 ts_tablet_manager.cc:1918] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:23.784477 3630 log.cc:1196] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/e11105ef9aea44668a5763f8977275a4
I20260926 06:57:23.784687 3630 ts_tablet_manager.cc:1939] T e11105ef9aea44668a5763f8977275a4 P 7b751efd090145f982765e75c0980b30: Deleting consensus metadata
I20260926 06:57:23.784961 3629 ts_tablet_manager.cc:1918] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:23.785015 3629 log.cc:1196] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/e11105ef9aea44668a5763f8977275a4
I20260926 06:57:23.785014 3047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 06:57:23.785167 3629 ts_tablet_manager.cc:1939] T e11105ef9aea44668a5763f8977275a4 P 9899c8b5a59741e9a704dd923f5f07b8: Deleting consensus metadata
W20260926 06:57:23.785171 3047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:23.785405 3031 catalog_manager.cc:4841] TS 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67:44219): tablet e11105ef9aea44668a5763f8977275a4 (table test-table [id=5887dce8104549c69727ca9aa4adacad]) successfully deleted
I20260926 06:57:23.785949 3034 catalog_manager.cc:4841] TS 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619): tablet e11105ef9aea44668a5763f8977275a4 (table test-table [id=5887dce8104549c69727ca9aa4adacad]) successfully deleted
I20260926 06:57:23.786343 3184 tablet_service.cc:1467] Processing CreateTablet for tablet 6e81176f885946eca4c0a5c67f9a2080 (DEFAULT_TABLE table=test-table [id=087efb386de446ed92641e6405315e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:23.786365 3464 tablet_service.cc:1467] Processing CreateTablet for tablet 6e81176f885946eca4c0a5c67f9a2080 (DEFAULT_TABLE table=test-table [id=087efb386de446ed92641e6405315e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:23.786453 3464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e81176f885946eca4c0a5c67f9a2080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:23.786455 3184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e81176f885946eca4c0a5c67f9a2080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:23.787283 3643 tablet_bootstrap.cc:492] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: Bootstrap starting.
I20260926 06:57:23.787319 3324 tablet_service.cc:1467] Processing CreateTablet for tablet 6e81176f885946eca4c0a5c67f9a2080 (DEFAULT_TABLE table=test-table [id=087efb386de446ed92641e6405315e80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:23.787418 3324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e81176f885946eca4c0a5c67f9a2080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:23.787595 3643 tablet_bootstrap.cc:654] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:23.787834 3641 tablet_bootstrap.cc:492] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c: Bootstrap starting.
I20260926 06:57:23.788142 3641 tablet_bootstrap.cc:654] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:23.788721 3641 tablet_bootstrap.cc:492] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c: No bootstrap required, opened a new log
I20260926 06:57:23.788724 3640 tablet_bootstrap.cc:492] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: Bootstrap starting.
I20260926 06:57:23.788771 3641 ts_tablet_manager.cc:1397] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:23.788894 3641 raft_consensus.cc:348] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.788971 3641 raft_consensus.cc:374] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:23.788996 3641 raft_consensus.cc:729] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aefd9600b7840379c52972e0f92157c, State: Initialized, Role: FOLLOWER
I20260926 06:57:23.789047 3641 consensus_queue.cc:260] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.789106 3640 tablet_bootstrap.cc:654] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:23.789141 3641 ts_tablet_manager.cc:1428] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:23.789419 3643 tablet_bootstrap.cc:492] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: No bootstrap required, opened a new log
I20260926 06:57:23.789471 3643 ts_tablet_manager.cc:1397] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:23.789601 3643 raft_consensus.cc:348] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.789695 3643 raft_consensus.cc:374] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:23.789719 3643 raft_consensus.cc:729] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899c8b5a59741e9a704dd923f5f07b8, State: Initialized, Role: FOLLOWER
I20260926 06:57:23.789716 3640 tablet_bootstrap.cc:492] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: No bootstrap required, opened a new log
I20260926 06:57:23.789757 3640 ts_tablet_manager.cc:1397] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:23.789768 3643 consensus_queue.cc:260] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.789845 3643 ts_tablet_manager.cc:1428] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:23.789913 3640 raft_consensus.cc:348] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.790000 3640 raft_consensus.cc:374] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:23.790021 3640 raft_consensus.cc:729] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b751efd090145f982765e75c0980b30, State: Initialized, Role: FOLLOWER
I20260926 06:57:23.790071 3640 consensus_queue.cc:260] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.790174 3640 ts_tablet_manager.cc:1428] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:23.832701 3624 raft_consensus.cc:482] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:23.832815 3624 raft_consensus.cc:504] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.833001 3624 leader_election.cc:288] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885), 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619)
I20260926 06:57:23.833205 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e81176f885946eca4c0a5c67f9a2080" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b751efd090145f982765e75c0980b30" is_pre_election: true
I20260926 06:57:23.833209 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e81176f885946eca4c0a5c67f9a2080" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aefd9600b7840379c52972e0f92157c" is_pre_election: true
I20260926 06:57:23.833302 3344 raft_consensus.cc:2454] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 0.
I20260926 06:57:23.833308 3204 raft_consensus.cc:2454] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 0.
I20260926 06:57:23.833444 3418 leader_election.cc:302] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b751efd090145f982765e75c0980b30, 9899c8b5a59741e9a704dd923f5f07b8; no voters:
I20260926 06:57:23.833539 3624 raft_consensus.cc:2781] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:23.833593 3624 raft_consensus.cc:482] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:23.833622 3624 raft_consensus.cc:3037] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:23.834304 3624 raft_consensus.cc:504] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.834434 3624 leader_election.cc:288] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 election: Requested vote from peers 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885), 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619)
I20260926 06:57:23.834611 3344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e81176f885946eca4c0a5c67f9a2080" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b751efd090145f982765e75c0980b30"
I20260926 06:57:23.834686 3344 raft_consensus.cc:3037] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:23.835410 3344 raft_consensus.cc:2454] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 1.
I20260926 06:57:23.835390 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e81176f885946eca4c0a5c67f9a2080" candidate_uuid: "9899c8b5a59741e9a704dd923f5f07b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aefd9600b7840379c52972e0f92157c"
I20260926 06:57:23.835479 3204 raft_consensus.cc:3037] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:23.835601 3418 leader_election.cc:302] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b751efd090145f982765e75c0980b30, 9899c8b5a59741e9a704dd923f5f07b8; no voters:
I20260926 06:57:23.835733 3624 raft_consensus.cc:2781] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:23.835783 3624 raft_consensus.cc:686] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 LEADER]: Becoming Leader. State: Replica: 9899c8b5a59741e9a704dd923f5f07b8, State: Running, Role: LEADER
I20260926 06:57:23.835824 3624 consensus_queue.cc:237] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } }
I20260926 06:57:23.836359 3204 raft_consensus.cc:2454] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9899c8b5a59741e9a704dd923f5f07b8 in term 1.
I20260926 06:57:23.836416 3047 catalog_manager.cc:5495] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "9899c8b5a59741e9a704dd923f5f07b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9899c8b5a59741e9a704dd923f5f07b8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 44219 } health_report { overall_health: HEALTHY } } }
I20260926 06:57:23.853339 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:23.850628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.1.74.67:44219)
0926 06:57:23.850664 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:23.850666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:23.850700 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:57:23.850774 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:23.850775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:23.850785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:23.850828 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:23.850988 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.851541 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.852579 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:23.852802 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:23.852888 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:23.853070 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:23.853099 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:23.853143 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:23.853226 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 06:57:23.853248 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:57:23.855509 3344 raft_consensus.cc:1261] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Refusing update from remote peer 9899c8b5a59741e9a704dd923f5f07b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:23.855610 3204 raft_consensus.cc:1261] T 6e81176f885946eca4c0a5c67f9a2080 P 5aefd9600b7840379c52972e0f92157c [term 1 FOLLOWER]: Refusing update from remote peer 9899c8b5a59741e9a704dd923f5f07b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:23.855867 3624 consensus_queue.cc:1035] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aefd9600b7840379c52972e0f92157c" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:23.855993 3560 consensus_queue.cc:1035] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b751efd090145f982765e75c0980b30" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:23.857337 3611 mvcc.cc:205] Tried to move back new op lower bound from 7333502336430764032 to 7333502336352022528. Current Snapshot: MvccSnapshot[applied={T|T < 7333502336430764032}]
W20260926 06:57:23.861131 3611 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 06:57:25.380916 3045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:25.381053 3045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:25.381893 3045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21aff96afdf445e1b628b50d61146aa6: Sending DeleteTablet for 3 replicas of tablet 6e81176f885946eca4c0a5c67f9a2080
I20260926 06:57:25.382136 3464 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e81176f885946eca4c0a5c67f9a2080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:25 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57762
I20260926 06:57:25.382180 3184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e81176f885946eca4c0a5c67f9a2080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:25 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:38326
I20260926 06:57:25.382272 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 3119
I20260926 06:57:25.382342 3324 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e81176f885946eca4c0a5c67f9a2080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:25 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57342
I20260926 06:57:25.382965 3695 tablet_replica.cc:330] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: stopping tablet replica
I20260926 06:57:25.383033 3695 raft_consensus.cc:2229] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:25.383091 3695 raft_consensus.cc:2258] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:25.383378 3695 ts_tablet_manager.cc:1905] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:25.384429 3693 tablet_replica.cc:330] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: stopping tablet replica
I20260926 06:57:25.384505 3693 raft_consensus.cc:2229] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:25.384617 3693 raft_consensus.cc:2258] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:25.384869 3693 ts_tablet_manager.cc:1905] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:25.385068 3695 ts_tablet_manager.cc:1918] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:57:25.385135 3695 log.cc:1196] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/6e81176f885946eca4c0a5c67f9a2080
I20260926 06:57:25.385370 3695 ts_tablet_manager.cc:1939] T 6e81176f885946eca4c0a5c67f9a2080 P 7b751efd090145f982765e75c0980b30: Deleting consensus metadata
I20260926 06:57:25.385707 3034 catalog_manager.cc:4841] TS 7b751efd090145f982765e75c0980b30 (127.1.74.66:34619): tablet 6e81176f885946eca4c0a5c67f9a2080 (table test-table [id=087efb386de446ed92641e6405315e80]) successfully deleted
I20260926 06:57:25.386668 3693 ts_tablet_manager.cc:1918] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:57:25.386718 3693 log.cc:1196] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/6e81176f885946eca4c0a5c67f9a2080
I20260926 06:57:25.386946 3693 ts_tablet_manager.cc:1939] T 6e81176f885946eca4c0a5c67f9a2080 P 9899c8b5a59741e9a704dd923f5f07b8: Deleting consensus metadata
I20260926 06:57:25.387245 3031 catalog_manager.cc:4841] TS 9899c8b5a59741e9a704dd923f5f07b8 (127.1.74.67:44219): tablet 6e81176f885946eca4c0a5c67f9a2080 (table test-table [id=087efb386de446ed92641e6405315e80]) successfully deleted
W20260926 06:57:25.390561 3034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:40885 (error 108)
I20260926 06:57:25.390929 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 3259
I20260926 06:57:25.391646 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:25.390960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40885 (local address 127.0.0.1:41474)
0926 06:57:25.391561 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:25.391611 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":65,"threads_started":1}
W20260926 06:57:25.391770 3034 catalog_manager.cc:4584] TS 5aefd9600b7840379c52972e0f92157c (127.1.74.65:40885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e81176f885946eca4c0a5c67f9a2080: Network error: Client connection negotiation failed: client connection to 127.1.74.65:40885: connect: Connection refused (error 111)
I20260926 06:57:25.397550 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 3399
I20260926 06:57:25.402667 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 3015
2026-09-26T06:57:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5377 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:25 dist-test-slave-sfvt krb5kdc[3700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:25 dist-test-slave-sfvt krb5kdc[3700](info): set up 2 sockets
Sep 26 06:57:25 dist-test-slave-sfvt krb5kdc[3700](info): commencing operation
krb5kdc: starting...
W20260926 06:57:26.455093 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:26 dist-test-slave-sfvt krb5kdc[3700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:26.573894 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36839
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:40427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:26.654747 3713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:26.654959 3713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:26.654981 3713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:26.656347 3713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:26.656420 3713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:26.656445 3713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:26.656463 3713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:26.656482 3713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:26.658154 3713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:40427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:36839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:26.658435 3713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:26.658690 3713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:26.663508 3718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:26.663508 3719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:26.663542 3721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:26.663878 3713 server_base.cc:1034] running on GCE node
I20260926 06:57:26.664088 3713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:26.664379 3713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:26.665535 3713 hybrid_clock.cc:648] HybridClock initialized: now 1790405846665522 us; error 27 us; skew 500 ppm
Sep 26 06:57:26 dist-test-slave-sfvt krb5kdc[3700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405846, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:26.668177 3713 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:26.668672 3713 webserver.cc:460] Webserver started at http://127.1.74.126:41415/ using document root <none> and password file <none>
I20260926 06:57:26.668900 3713 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:26.668942 3713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:26.669030 3713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:26.670080 3713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "7c0920a1f8e34a7a9976f0d63cc24c09"
format_stamp: "Formatted at 2026-09-26 06:57:26 on dist-test-slave-sfvt"
server_key: "8a999e3e6a479a1c217ef9f2fb97fdcf"
server_key_iv: "c694eedbff2ea274399cb6e1057998fd"
server_key_version: "encryptionkey@0"
I20260926 06:57:26.670380 3713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "7c0920a1f8e34a7a9976f0d63cc24c09"
format_stamp: "Formatted at 2026-09-26 06:57:26 on dist-test-slave-sfvt"
server_key: "8a999e3e6a479a1c217ef9f2fb97fdcf"
server_key_iv: "c694eedbff2ea274399cb6e1057998fd"
server_key_version: "encryptionkey@0"
I20260926 06:57:26.671672 3713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:26.672400 3728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:26.672794 3713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:26.672881 3713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "7c0920a1f8e34a7a9976f0d63cc24c09"
format_stamp: "Formatted at 2026-09-26 06:57:26 on dist-test-slave-sfvt"
server_key: "8a999e3e6a479a1c217ef9f2fb97fdcf"
server_key_iv: "c694eedbff2ea274399cb6e1057998fd"
server_key_version: "encryptionkey@0"
I20260926 06:57:26.672936 3713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:26.686589 3713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:26.688954 3713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:26.689064 3713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:26.692806 3713 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:36839
I20260926 06:57:26.692873 3780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:36839 every 8 connection(s)
I20260926 06:57:26.693238 3713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:26.694144 3781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:26.696442 3781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09: Bootstrap starting.
I20260926 06:57:26.697012 3781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:26.697307 3781 log.cc:824] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:26.697954 3781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09: No bootstrap required, opened a new log
I20260926 06:57:26.698782 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 3713
I20260926 06:57:26.698848 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:26.699046 1321 external_mini_cluster.cc:1428] Setting key a0b3b414406db0360b54d3d8d1bdd7e5
I20260926 06:57:26.700080 3781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0920a1f8e34a7a9976f0d63cc24c09" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36839 } }
I20260926 06:57:26.700263 3781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:26.700301 3781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0920a1f8e34a7a9976f0d63cc24c09, State: Initialized, Role: FOLLOWER
I20260926 06:57:26.700451 3781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0920a1f8e34a7a9976f0d63cc24c09" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36839 } }
I20260926 06:57:26.700516 3781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:26.700537 3781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:26.700608 3781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:26.701668 3781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0920a1f8e34a7a9976f0d63cc24c09" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36839 } }
I20260926 06:57:26.701772 3781 leader_election.cc:302] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c0920a1f8e34a7a9976f0d63cc24c09; no voters:
I20260926 06:57:26.701923 3781 leader_election.cc:288] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:26.701984 3786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:26.702148 3781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:26.702154 3786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0920a1f8e34a7a9976f0d63cc24c09, State: Running, Role: LEADER
I20260926 06:57:26.702304 3786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0920a1f8e34a7a9976f0d63cc24c09" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36839 } }
I20260926 06:57:26.702673 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c0920a1f8e34a7a9976f0d63cc24c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0920a1f8e34a7a9976f0d63cc24c09" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36839 } } }
I20260926 06:57:26.702841 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:26.702669 3788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c0920a1f8e34a7a9976f0d63cc24c09. Latest consensus state: current_term: 1 leader_uuid: "7c0920a1f8e34a7a9976f0d63cc24c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0920a1f8e34a7a9976f0d63cc24c09" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36839 } } }
I20260926 06:57:26.703140 3788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09 [sys.catalog]: This master's current role is: LEADER
W20260926 06:57:26.704414 3801 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:26.704473 3801 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:26.704571 3795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:26.704701 3795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 06:57:26 dist-test-slave-sfvt krb5kdc[3700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:26.707070 3795 catalog_manager.cc:1347] Generated new cluster ID: e0d9f2c76b2c4051b6fb2c6872e1b887
I20260926 06:57:26.707124 3795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:26.710302 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:26.699618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.1.74.126:36839)
0926 06:57:26.699822 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:26.699831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:26.699870 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:57:26.700041 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:26.700048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:26.700106 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:26.700254 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:26.701034 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:26.702478 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:26.703576 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:26.704131 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:26.704274 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:26.707024 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:26.707027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:26.707028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:26.707037 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:26.708795 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:26.709168 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:26.709171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:26.709171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:26.709275 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:26.709508 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:26.709509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:26.709509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:26.709745 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:26.709933 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:26.709998 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 06:57:26.710135 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260926 06:57:26.716823 3795 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:26.717353 3795 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:26.727277 3795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c0920a1f8e34a7a9976f0d63cc24c09: Generated new TSK 0
I20260926 06:57:26.727500 3795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:26.763391 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36839
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:40427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:26.844133 3807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:26.844372 3807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:26.844409 3807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:26.845557 3807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:26.845629 3807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:26.845674 3807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:26.847112 3807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:40427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:26.847399 3807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:26.847632 3807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:26.852483 3812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:26.852487 3815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:26.852725 3813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:26.852779 3807 server_base.cc:1034] running on GCE node
I20260926 06:57:26.852985 3807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:26.853224 3807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:26.854384 3807 hybrid_clock.cc:648] HybridClock initialized: now 1790405846854375 us; error 37 us; skew 500 ppm
Sep 26 06:57:26 dist-test-slave-sfvt krb5kdc[3700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405846, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:26.856824 3807 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:26.857268 3807 webserver.cc:460] Webserver started at http://127.1.74.65:35717/ using document root <none> and password file <none>
I20260926 06:57:26.857448 3807 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:26.857487 3807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:26.857577 3807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:26.858559 3807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "26e1187c111543628cbaebb71d396b6d"
format_stamp: "Formatted at 2026-09-26 06:57:26 on dist-test-slave-sfvt"
server_key: "fffe295c61cf4a6a3a77a994a9b6dbc2"
server_key_iv: "cac6969c356d1f4b29934d2ab5dc0731"
server_key_version: "encryptionkey@0"
I20260926 06:57:26.858853 3807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "26e1187c111543628cbaebb71d396b6d"
format_stamp: "Formatted at 2026-09-26 06:57:26 on dist-test-slave-sfvt"
server_key: "fffe295c61cf4a6a3a77a994a9b6dbc2"
server_key_iv: "cac6969c356d1f4b29934d2ab5dc0731"
server_key_version: "encryptionkey@0"
I20260926 06:57:26.860209 3807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:26.860924 3822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:26.861088 3807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:26.861158 3807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "26e1187c111543628cbaebb71d396b6d"
format_stamp: "Formatted at 2026-09-26 06:57:26 on dist-test-slave-sfvt"
server_key: "fffe295c61cf4a6a3a77a994a9b6dbc2"
server_key_iv: "cac6969c356d1f4b29934d2ab5dc0731"
server_key_version: "encryptionkey@0"
I20260926 06:57:26.861210 3807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:26.870785 3807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:26.872555 3807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:26.872663 3807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:26.872824 3807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:26.873073 3807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:26.873103 3807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:26.873124 3807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:26.873136 3807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:26.879099 3807 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:40029
I20260926 06:57:26.879140 3935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:40029 every 8 connection(s)
I20260926 06:57:26.879545 3807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:57:26 dist-test-slave-sfvt krb5kdc[3700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405846, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:26.888182 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:26.880247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:40749 (local address 127.1.74.126:36839)
0926 06:57:26.880310 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:26.880313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:26.880518 (+ 205us) server_negotiation.cc:407] Connection header received
0926 06:57:26.880988 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:26.880989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:26.881001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:26.881058 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:26.881934 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:26.882421 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:26.883365 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:26.883559 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:26.883614 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:26.885612 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:26.885614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:26.885614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:26.885621 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:26.887022 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:26.887432 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:26.887433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:26.887433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:26.887480 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:26.887669 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:26.887670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:26.887670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:26.887759 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:26.887803 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:26.888039 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 06:57:26.888101 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:57:26.888372 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:26.880222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36839 (local address 127.1.74.65:40749)
0926 06:57:26.880394 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:26.880435 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:26.880903 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:26.881121 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:26.881124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:26.881679 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:26.881863 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:26.881867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:26.882495 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:26.882496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:26.883292 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:26.883294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:26.883630 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:26.883631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:26.883724 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:26.884241 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:26.884249 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:26.885527 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:26.887095 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:26.887100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:26.887102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:26.887373 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:26.887539 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:26.887539 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:26.887540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:26.887631 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:26.887808 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:26.887812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:26.887930 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 06:57:26.888010 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:57:26.888122 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260926 06:57:26.888352 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 3807
I20260926 06:57:26.888451 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:26.888638 1321 external_mini_cluster.cc:1428] Setting key d5d403764be56040105d83be839cf1e8
I20260926 06:57:26.888846 3936 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36839
I20260926 06:57:26.888963 3936 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:26.889204 3936 heartbeater.cc:511] Master 127.1.74.126:36839 requested a full tablet report, sending...
I20260926 06:57:26.889748 3745 ts_manager.cc:194] Registered new tserver with Master: 26e1187c111543628cbaebb71d396b6d (127.1.74.65:40029)
I20260926 06:57:26.890678 3745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:40749
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:26.914166 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36839
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:40427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:26.995635 3941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:26.995888 3941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:26.995929 3941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:26.997099 3941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:26.997170 3941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:26.997211 3941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:26.998629 3941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:40427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:26.999096 3941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:26.999401 3941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:27.004895 3946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:27.004914 3947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:27.004895 3949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:27.004949 3941 server_base.cc:1034] running on GCE node
I20260926 06:57:27.005204 3941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:27.005390 3941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:27.006536 3941 hybrid_clock.cc:648] HybridClock initialized: now 1790405847006525 us; error 25 us; skew 500 ppm
Sep 26 06:57:27 dist-test-slave-sfvt krb5kdc[3700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:27.009095 3941 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:27.009522 3941 webserver.cc:460] Webserver started at http://127.1.74.66:39493/ using document root <none> and password file <none>
I20260926 06:57:27.009754 3941 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:27.009799 3941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:27.009946 3941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:27.010982 3941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "eb4658f03c884cf981ba8a781b9c5e70"
format_stamp: "Formatted at 2026-09-26 06:57:27 on dist-test-slave-sfvt"
server_key: "9fe96a8973138983cdd5543b4d4b82e0"
server_key_iv: "062f24418af401d6070307bef974d9d1"
server_key_version: "encryptionkey@0"
I20260926 06:57:27.011379 3941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "eb4658f03c884cf981ba8a781b9c5e70"
format_stamp: "Formatted at 2026-09-26 06:57:27 on dist-test-slave-sfvt"
server_key: "9fe96a8973138983cdd5543b4d4b82e0"
server_key_iv: "062f24418af401d6070307bef974d9d1"
server_key_version: "encryptionkey@0"
I20260926 06:57:27.012642 3941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:27.013353 3956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:27.013526 3941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:27.013622 3941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "eb4658f03c884cf981ba8a781b9c5e70"
format_stamp: "Formatted at 2026-09-26 06:57:27 on dist-test-slave-sfvt"
server_key: "9fe96a8973138983cdd5543b4d4b82e0"
server_key_iv: "062f24418af401d6070307bef974d9d1"
server_key_version: "encryptionkey@0"
I20260926 06:57:27.013707 3941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:27.035234 3941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:27.037583 3941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:27.037748 3941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:27.037989 3941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:27.038300 3941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:27.038353 3941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:27.038395 3941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:27.038424 3941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:27.044100 3941 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:43539
I20260926 06:57:27.044144 4069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:43539 every 8 connection(s)
I20260926 06:57:27.044494 3941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:27.048812 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 3941
I20260926 06:57:27.048916 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:27.049119 1321 external_mini_cluster.cc:1428] Setting key b5c340a35939a3a9e7ff7e116761a8ca
Sep 26 06:57:27 dist-test-slave-sfvt krb5kdc[3700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:27.054563 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:27.045207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36839 (local address 127.1.74.66:48979)
0926 06:57:27.045397 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:27.045446 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:27.045950 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:27.046193 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:27.046197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:27.046753 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:27.046948 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:27.046952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:27.047739 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:27.047740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:27.048538 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:27.048541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:27.049075 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:27.049076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:27.049232 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:27.049817 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:27.049827 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:27.051168 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:27.053111 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:27.053116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:27.053119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:27.053437 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:27.053661 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:27.053662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:27.053662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:27.053770 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:27.054053 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:27.054058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:27.054175 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 06:57:27.054283 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 06:57:27.054401 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260926 06:57:27.054577 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:27.045254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:48979 (local address 127.1.74.126:36839)
0926 06:57:27.045342 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:27.045344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:27.045529 (+ 185us) server_negotiation.cc:407] Connection header received
0926 06:57:27.046063 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:27.046064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:27.046076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:27.046123 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:27.047059 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:27.047641 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:27.048647 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:27.048983 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:27.049055 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:27.051282 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:27.051283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:27.051283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:27.051291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:27.053023 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:27.053519 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:27.053521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:27.053522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:27.053587 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:27.053837 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:27.053838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:27.053838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:27.053979 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:27.054051 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:27.054333 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 06:57:27.054441 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:57:27.055095 4070 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36839
I20260926 06:57:27.055202 4070 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:27.055428 4070 heartbeater.cc:511] Master 127.1.74.126:36839 requested a full tablet report, sending...
I20260926 06:57:27.055788 3745 ts_manager.cc:194] Registered new tserver with Master: eb4658f03c884cf981ba8a781b9c5e70 (127.1.74.66:43539)
I20260926 06:57:27.056149 3745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:48979
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:27.075030 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36839
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:40427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:27.154362 4075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:27.154595 4075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:27.154633 4075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:27.155867 4075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:27.155932 4075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:27.155982 4075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:27.157411 4075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:40427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:27.157680 4075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:27.157904 4075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:27.162798 4080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:27.162799 4083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:27.162863 4081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:27.163141 4075 server_base.cc:1034] running on GCE node
I20260926 06:57:27.163383 4075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:27.163646 4075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:27.164783 4075 hybrid_clock.cc:648] HybridClock initialized: now 1790405847164765 us; error 37 us; skew 500 ppm
Sep 26 06:57:27 dist-test-slave-sfvt krb5kdc[3700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:27.167305 4075 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:27.167754 4075 webserver.cc:460] Webserver started at http://127.1.74.67:46269/ using document root <none> and password file <none>
I20260926 06:57:27.167995 4075 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:27.168041 4075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:27.168169 4075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:27.169201 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "4c97234219a6400abc4add5669dc9b2c"
format_stamp: "Formatted at 2026-09-26 06:57:27 on dist-test-slave-sfvt"
server_key: "77304cd2c7f8799d985924a8c2657aa9"
server_key_iv: "3db3988e098358ef729b77684e2a2cc4"
server_key_version: "encryptionkey@0"
I20260926 06:57:27.169544 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "4c97234219a6400abc4add5669dc9b2c"
format_stamp: "Formatted at 2026-09-26 06:57:27 on dist-test-slave-sfvt"
server_key: "77304cd2c7f8799d985924a8c2657aa9"
server_key_iv: "3db3988e098358ef729b77684e2a2cc4"
server_key_version: "encryptionkey@0"
I20260926 06:57:27.170979 4075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:27.172016 4090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:27.172199 4075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:27.172291 4075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "4c97234219a6400abc4add5669dc9b2c"
format_stamp: "Formatted at 2026-09-26 06:57:27 on dist-test-slave-sfvt"
server_key: "77304cd2c7f8799d985924a8c2657aa9"
server_key_iv: "3db3988e098358ef729b77684e2a2cc4"
server_key_version: "encryptionkey@0"
I20260926 06:57:27.172367 4075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:27.199196 4075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:27.201392 4075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:27.201550 4075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:27.201797 4075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:27.202111 4075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:27.202167 4075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:27.202209 4075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:27.202237 4075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:27.208325 4075 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:34417
I20260926 06:57:27.208366 4203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:34417 every 8 connection(s)
I20260926 06:57:27.208722 4075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:57:27.209689 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 4075
I20260926 06:57:27.209888 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:27.210090 1321 external_mini_cluster.cc:1428] Setting key 5d1a66f8edd253b7b2730e82e84f5083
Sep 26 06:57:27 dist-test-slave-sfvt krb5kdc[3700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405847, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:27.218410 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:27.209466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:38071 (local address 127.1.74.126:36839)
0926 06:57:27.209571 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:27.209573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:27.209741 (+ 168us) server_negotiation.cc:407] Connection header received
0926 06:57:27.210446 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:27.210447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:27.210458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:27.210516 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:27.211617 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:27.212100 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:27.213160 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:27.213380 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:27.213440 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:27.215589 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:27.215591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:27.215591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:27.215599 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:27.216970 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:27.217384 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:27.217385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:27.217386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:27.217488 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:27.217765 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:27.217766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:27.217767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:27.217885 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:27.217961 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:27.218237 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 06:57:27.218314 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 06:57:27.218613 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:27.209434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36839 (local address 127.1.74.67:38071)
0926 06:57:27.209603 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:27.209646 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:27.210258 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:27.210597 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:27.210601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:27.211245 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:27.211511 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:27.211516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:27.212207 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:27.212208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:27.213063 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:27.213066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:27.213472 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:27.213473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:27.213584 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:27.214014 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:27.214023 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:27.215463 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:27.217057 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:27.217063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:27.217065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:27.217295 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:27.217563 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:27.217564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:27.217564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:27.217688 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:27.217952 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:27.217958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:27.218098 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 06:57:27.218215 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 06:57:27.218376 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260926 06:57:27.219098 4204 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36839
I20260926 06:57:27.219177 4204 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:27.219375 4204 heartbeater.cc:511] Master 127.1.74.126:36839 requested a full tablet report, sending...
I20260926 06:57:27.219725 3745 ts_manager.cc:194] Registered new tserver with Master: 4c97234219a6400abc4add5669dc9b2c (127.1.74.67:34417)
I20260926 06:57:27.220054 3745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:38071
I20260926 06:57:27.221870 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 06:57:27.232903 3785 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:27.231905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.1.74.126:36839)
0926 06:57:27.231996 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:27.231998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:27.232088 (+ 90us) server_negotiation.cc:407] Connection header received
0926 06:57:27.232177 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:27.232178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:27.232189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:27.232240 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:27.232822 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59454: BlockingRecv error: recv got EOF from 127.0.0.1:59454 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 06:57:27.233644 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 3807
I20260926 06:57:27.237852 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 3941
I20260926 06:57:27.241878 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 4075
I20260926 06:57:27.245973 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 3713
2026-09-26T06:57:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1841 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:27 dist-test-slave-sfvt krb5kdc[4216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:27 dist-test-slave-sfvt krb5kdc[4216](info): set up 2 sockets
Sep 26 06:57:27 dist-test-slave-sfvt krb5kdc[4216](info): commencing operation
krb5kdc: starting...
W20260926 06:57:28.287384 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:28 dist-test-slave-sfvt krb5kdc[4216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:28.411027 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:40097
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:40097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:28.490571 4229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:28.490833 4229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:28.490870 4229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:28.492101 4229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:28.492165 4229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:28.492188 4229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:28.492209 4229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:28.492237 4229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:28.493698 4229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:40097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:28.493969 4229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:28.494181 4229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:28.499135 4237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:28.499202 4235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:28.499218 4234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:28.499401 4229 server_base.cc:1034] running on GCE node
I20260926 06:57:28.499678 4229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:28.499965 4229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:28.501121 4229 hybrid_clock.cc:648] HybridClock initialized: now 1790405848501109 us; error 25 us; skew 500 ppm
Sep 26 06:57:28 dist-test-slave-sfvt krb5kdc[4216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405848, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:28.503525 4229 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:28.503942 4229 webserver.cc:460] Webserver started at http://127.1.74.126:36747/ using document root <none> and password file <none>
I20260926 06:57:28.504174 4229 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:28.504242 4229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:28.504391 4229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:28.505462 4229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "b2764cbf15c3445c8b99acf3c6449989"
format_stamp: "Formatted at 2026-09-26 06:57:28 on dist-test-slave-sfvt"
server_key: "54505059346e9bd9bc8f7cbcaf9402ba"
server_key_iv: "c10c9dbde56b4e039ec45d852f7aa415"
server_key_version: "encryptionkey@0"
I20260926 06:57:28.505810 4229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "b2764cbf15c3445c8b99acf3c6449989"
format_stamp: "Formatted at 2026-09-26 06:57:28 on dist-test-slave-sfvt"
server_key: "54505059346e9bd9bc8f7cbcaf9402ba"
server_key_iv: "c10c9dbde56b4e039ec45d852f7aa415"
server_key_version: "encryptionkey@0"
I20260926 06:57:28.507158 4229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:28.507972 4244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:28.508150 4229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:28.508248 4229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "b2764cbf15c3445c8b99acf3c6449989"
format_stamp: "Formatted at 2026-09-26 06:57:28 on dist-test-slave-sfvt"
server_key: "54505059346e9bd9bc8f7cbcaf9402ba"
server_key_iv: "c10c9dbde56b4e039ec45d852f7aa415"
server_key_version: "encryptionkey@0"
I20260926 06:57:28.508329 4229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:28.517414 4229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:28.519448 4229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:28.519598 4229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:28.523222 4229 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:40097
I20260926 06:57:28.523300 4296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:40097 every 8 connection(s)
I20260926 06:57:28.523641 4229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:28.524617 4297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:28.525586 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 4229
I20260926 06:57:28.525691 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:28.525859 1321 external_mini_cluster.cc:1428] Setting key 7e7a7a731e44b1f396a5569685be2890
I20260926 06:57:28.527313 4297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989: Bootstrap starting.
I20260926 06:57:28.527957 4297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:28.528280 4297 log.cc:824] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:28.529011 4297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989: No bootstrap required, opened a new log
I20260926 06:57:28.531001 4297 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2764cbf15c3445c8b99acf3c6449989" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40097 } }
I20260926 06:57:28.531167 4297 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:28.531222 4297 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2764cbf15c3445c8b99acf3c6449989, State: Initialized, Role: FOLLOWER
I20260926 06:57:28.531440 4297 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2764cbf15c3445c8b99acf3c6449989" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40097 } }
I20260926 06:57:28.531548 4297 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:28.531596 4297 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:28.531648 4297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 06:57:28 dist-test-slave-sfvt krb5kdc[4216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:28.532223 4297 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2764cbf15c3445c8b99acf3c6449989" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40097 } }
I20260926 06:57:28.532353 4297 leader_election.cc:302] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2764cbf15c3445c8b99acf3c6449989; no voters:
I20260926 06:57:28.532569 4297 leader_election.cc:288] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:28.532601 4302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:28.532716 4302 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [term 1 LEADER]: Becoming Leader. State: Replica: b2764cbf15c3445c8b99acf3c6449989, State: Running, Role: LEADER
I20260926 06:57:28.532810 4302 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2764cbf15c3445c8b99acf3c6449989" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40097 } }
I20260926 06:57:28.532836 4297 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:28.533456 4304 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2764cbf15c3445c8b99acf3c6449989. Latest consensus state: current_term: 1 leader_uuid: "b2764cbf15c3445c8b99acf3c6449989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2764cbf15c3445c8b99acf3c6449989" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40097 } } }
I20260926 06:57:28.533550 4304 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:28.533783 4308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:28.533730 4303 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2764cbf15c3445c8b99acf3c6449989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2764cbf15c3445c8b99acf3c6449989" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40097 } } }
I20260926 06:57:28.533883 4303 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:28.534302 4308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:28.536021 4319 mvcc.cc:205] Tried to move back new op lower bound from 7333502355600564224 to 7333502355591004160. Current Snapshot: MvccSnapshot[applied={T|T < 7333502355600564224}]
I20260926 06:57:28.536808 4308 catalog_manager.cc:1347] Generated new cluster ID: 09de209d44de4ef8a55de55c365d8dab
I20260926 06:57:28.536849 4308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:28.537292 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:28.526364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.1.74.126:40097)
0926 06:57:28.526564 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:28.526571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:28.526596 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:57:28.527068 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:28.527074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:28.527119 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:28.527267 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:28.527926 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:28.528801 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:28.529627 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:28.530044 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:28.530164 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:28.532752 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:28.532756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:28.532756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:28.532768 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:28.535225 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:28.535660 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:28.535663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:28.535663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:28.535790 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:28.536007 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:28.536008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:28.536009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:28.536308 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:28.536424 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:28.536946 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 06:57:28.537081 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260926 06:57:28.542423 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:57:28.564572 4308 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:28.565150 4308 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:28.571753 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:28.564663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.1.74.126:40097)
0926 06:57:28.564741 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:28.564743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:28.567465 (+ 2722us) server_negotiation.cc:407] Connection header received
0926 06:57:28.567577 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:28.567578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:28.567589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:28.567630 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:28.568134 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:28.568928 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:28.570038 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:28.570352 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:28.570393 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:28.571209 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:28.571210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:28.571211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:28.571451 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:28.571471 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:28.571478 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:28.571512 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:28.571641 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 06:57:28.571670 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":58}
I20260926 06:57:28.575555 4308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2764cbf15c3445c8b99acf3c6449989: Generated new TSK 0
I20260926 06:57:28.575743 4308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:57:28.580708 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 4229
2026-09-26T06:57:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1332 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:28 dist-test-slave-sfvt krb5kdc[4330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:28 dist-test-slave-sfvt krb5kdc[4330](info): set up 2 sockets
Sep 26 06:57:28 dist-test-slave-sfvt krb5kdc[4330](info): commencing operation
krb5kdc: starting...
W20260926 06:57:29.615612 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:29 dist-test-slave-sfvt krb5kdc[4330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:29.734493 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36707
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:36707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:29.829960 4343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:29.830209 4343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:29.830245 4343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:29.831432 4343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:29.831498 4343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:29.831522 4343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:29.831543 4343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:29.831597 4343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:29.833091 4343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:36707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:29.833366 4343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:29.833603 4343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:29.838604 4349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:29.838654 4348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:29.838653 4351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:29.838825 4343 server_base.cc:1034] running on GCE node
I20260926 06:57:29.839105 4343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:29.839433 4343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:29.840595 4343 hybrid_clock.cc:648] HybridClock initialized: now 1790405849840600 us; error 48 us; skew 500 ppm
Sep 26 06:57:29 dist-test-slave-sfvt krb5kdc[4330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405849, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:29.843044 4343 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:29.843529 4343 webserver.cc:460] Webserver started at http://127.1.74.126:34993/ using document root <none> and password file <none>
I20260926 06:57:29.843770 4343 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:29.843842 4343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:29.843987 4343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:29.845017 4343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "1731e97d256d4feaa3b11f7a51000bfe"
format_stamp: "Formatted at 2026-09-26 06:57:29 on dist-test-slave-sfvt"
server_key: "1cfc68289a88866ae23a099fb0e4893c"
server_key_iv: "d089e4c9bea71ba6aff12e8d9970c887"
server_key_version: "encryptionkey@0"
I20260926 06:57:29.845372 4343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "1731e97d256d4feaa3b11f7a51000bfe"
format_stamp: "Formatted at 2026-09-26 06:57:29 on dist-test-slave-sfvt"
server_key: "1cfc68289a88866ae23a099fb0e4893c"
server_key_iv: "d089e4c9bea71ba6aff12e8d9970c887"
server_key_version: "encryptionkey@0"
I20260926 06:57:29.846684 4343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:29.847504 4358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:29.847707 4343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:29.847815 4343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "1731e97d256d4feaa3b11f7a51000bfe"
format_stamp: "Formatted at 2026-09-26 06:57:29 on dist-test-slave-sfvt"
server_key: "1cfc68289a88866ae23a099fb0e4893c"
server_key_iv: "d089e4c9bea71ba6aff12e8d9970c887"
server_key_version: "encryptionkey@0"
I20260926 06:57:29.847889 4343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:29.877131 4343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:29.879554 4343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:29.879729 4343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:29.883694 4343 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:36707
I20260926 06:57:29.883728 4410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:36707 every 8 connection(s)
I20260926 06:57:29.884086 4343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:29.885198 4411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:29.887482 4411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe: Bootstrap starting.
I20260926 06:57:29.888072 4411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:29.888357 4411 log.cc:824] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:29.889082 4411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe: No bootstrap required, opened a new log
I20260926 06:57:29.889686 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 4343
I20260926 06:57:29.889788 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:29.889966 1321 external_mini_cluster.cc:1428] Setting key 36d64202b0a2ac40c81023b59acea316
I20260926 06:57:29.891355 4411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731e97d256d4feaa3b11f7a51000bfe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36707 } }
I20260926 06:57:29.891530 4411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:29.891570 4411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1731e97d256d4feaa3b11f7a51000bfe, State: Initialized, Role: FOLLOWER
I20260926 06:57:29.891723 4411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731e97d256d4feaa3b11f7a51000bfe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36707 } }
I20260926 06:57:29.891816 4411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:29.891855 4411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:29.891906 4411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:29.892917 4411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731e97d256d4feaa3b11f7a51000bfe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36707 } }
I20260926 06:57:29.893006 4411 leader_election.cc:302] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1731e97d256d4feaa3b11f7a51000bfe; no voters:
I20260926 06:57:29.893218 4411 leader_election.cc:288] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:29.893237 4416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:29.893348 4416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [term 1 LEADER]: Becoming Leader. State: Replica: 1731e97d256d4feaa3b11f7a51000bfe, State: Running, Role: LEADER
I20260926 06:57:29.893429 4416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731e97d256d4feaa3b11f7a51000bfe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36707 } }
I20260926 06:57:29.893496 4411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:29.893751 4417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1731e97d256d4feaa3b11f7a51000bfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731e97d256d4feaa3b11f7a51000bfe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36707 } } }
I20260926 06:57:29.893976 4417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:29.893909 4418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1731e97d256d4feaa3b11f7a51000bfe. Latest consensus state: current_term: 1 leader_uuid: "1731e97d256d4feaa3b11f7a51000bfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731e97d256d4feaa3b11f7a51000bfe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36707 } } }
I20260926 06:57:29.894212 4418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:29.894608 4424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:29.895115 4424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:29.896737 4424 catalog_manager.cc:1347] Generated new cluster ID: e73bc77e1cbb4006813379ccc251a4cc
I20260926 06:57:29.896791 4424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:57:29 dist-test-slave-sfvt krb5kdc[4330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:29.901378 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:29.890475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.1.74.126:36707)
0926 06:57:29.890658 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:29.890665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:29.890693 (+ 28us) server_negotiation.cc:407] Connection header received
0926 06:57:29.891319 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:29.891335 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:29.891386 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:29.891504 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:29.892302 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:29.893380 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:29.894471 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:29.895116 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:29.895605 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:29.897921 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:29.897925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:29.897925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:29.897936 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:29.899811 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:29.900186 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:29.900189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:29.900189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:29.900302 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:29.900517 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:29.900518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:29.900518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:29.900767 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:29.900956 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:29.901036 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 06:57:29.901177 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260926 06:57:29.913079 4424 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:29.913838 4424 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:29.923600 4424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1731e97d256d4feaa3b11f7a51000bfe: Generated new TSK 0
I20260926 06:57:29.923820 4424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:29.956357 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36707
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:30.037207 4437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:30.037438 4437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:30.037477 4437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:30.038622 4437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:30.038687 4437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:30.038730 4437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:30.040197 4437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:30.040457 4437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:30.040681 4437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:30.045812 4445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:30.045818 4442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:30.045948 4437 server_base.cc:1034] running on GCE node
W20260926 06:57:30.045876 4443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:30.046211 4437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:30.046460 4437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:30.047619 4437 hybrid_clock.cc:648] HybridClock initialized: now 1790405850047617 us; error 35 us; skew 500 ppm
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:30.050130 4437 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:30.050638 4437 webserver.cc:460] Webserver started at http://127.1.74.65:42285/ using document root <none> and password file <none>
I20260926 06:57:30.050886 4437 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:30.050951 4437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:30.051084 4437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:30.052275 4437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "b98d5d8a54b74ad0adabf8a4f6ce194b"
format_stamp: "Formatted at 2026-09-26 06:57:30 on dist-test-slave-sfvt"
server_key: "b24549dee8b798cd7673ed5be0609bbf"
server_key_iv: "fc05609f7b34003069a7c4f1c096a44b"
server_key_version: "encryptionkey@0"
I20260926 06:57:30.052623 4437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "b98d5d8a54b74ad0adabf8a4f6ce194b"
format_stamp: "Formatted at 2026-09-26 06:57:30 on dist-test-slave-sfvt"
server_key: "b24549dee8b798cd7673ed5be0609bbf"
server_key_iv: "fc05609f7b34003069a7c4f1c096a44b"
server_key_version: "encryptionkey@0"
I20260926 06:57:30.053931 4437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:57:30.054746 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:30.054910 4437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:30.055009 4437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "b98d5d8a54b74ad0adabf8a4f6ce194b"
format_stamp: "Formatted at 2026-09-26 06:57:30 on dist-test-slave-sfvt"
server_key: "b24549dee8b798cd7673ed5be0609bbf"
server_key_iv: "fc05609f7b34003069a7c4f1c096a44b"
server_key_version: "encryptionkey@0"
I20260926 06:57:30.055080 4437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:30.070264 4437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:30.072778 4437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:30.072888 4437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:30.073056 4437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:30.073354 4437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:30.073385 4437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:30.073437 4437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:30.073462 4437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:30.078830 4437 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:44869
I20260926 06:57:30.078914 4565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:44869 every 8 connection(s)
I20260926 06:57:30.079231 4437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:30.081311 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 4437
I20260926 06:57:30.081387 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:30.081574 1321 external_mini_cluster.cc:1428] Setting key 986f63f4c29db2e75c59c771ca4ab195
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:30.089751 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:30.080076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:59529 (local address 127.1.74.126:36707)
0926 06:57:30.080200 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:30.080202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:30.080388 (+ 186us) server_negotiation.cc:407] Connection header received
0926 06:57:30.080862 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:30.080863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:30.080876 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:30.080930 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:30.082058 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.082758 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.083888 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.084221 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.084296 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:30.086792 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:30.086794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:30.086795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:30.086803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:30.088260 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:30.088761 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:30.088761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:30.088762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:30.088809 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:30.089081 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:30.089081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:30.089081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:30.089192 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:30.089250 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:30.089589 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 06:57:30.089678 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:57:30.090006 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:30.080046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36707 (local address 127.1.74.65:59529)
0926 06:57:30.080265 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:30.080302 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:30.080776 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:30.081006 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:30.081010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:30.081729 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:30.081982 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:30.081987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.082859 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.082860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:30.083804 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:30.083806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.084312 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.084313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:30.084409 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:30.084941 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:30.084949 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:30.086668 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:30.088348 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:30.088355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:30.088357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:30.088669 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:30.088875 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:30.088876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:30.088877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:30.089017 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:30.089267 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:30.089273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:30.089444 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 06:57:30.089572 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 06:57:30.089762 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
I20260926 06:57:30.090595 4566 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36707
I20260926 06:57:30.090708 4566 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:30.090888 4566 heartbeater.cc:511] Master 127.1.74.126:36707 requested a full tablet report, sending...
I20260926 06:57:30.091500 4375 ts_manager.cc:194] Registered new tserver with Master: b98d5d8a54b74ad0adabf8a4f6ce194b (127.1.74.65:44869)
I20260926 06:57:30.092106 4375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:59529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:30.107940 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36707
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:30.190442 4571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:30.190667 4571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:30.190691 4571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:30.192045 4571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:30.192113 4571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:30.192158 4571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:30.193760 4571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:30.194027 4571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:30.194316 4571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:30.199556 4577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:30.199540 4576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:30.199623 4571 server_base.cc:1034] running on GCE node
W20260926 06:57:30.199551 4579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:30.199941 4571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:30.200191 4571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:30.201349 4571 hybrid_clock.cc:648] HybridClock initialized: now 1790405850201339 us; error 30 us; skew 500 ppm
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:30.203877 4571 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:30.204315 4571 webserver.cc:460] Webserver started at http://127.1.74.66:44873/ using document root <none> and password file <none>
I20260926 06:57:30.204542 4571 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:30.204609 4571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:30.204761 4571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:30.205790 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "8b3706b13248460dbee8a8c74c5e3faa"
format_stamp: "Formatted at 2026-09-26 06:57:30 on dist-test-slave-sfvt"
server_key: "df59d0d7dd15944bb600f5a4424172c7"
server_key_iv: "0e38544cd20d92fb859bd7a1d0d5ed53"
server_key_version: "encryptionkey@0"
I20260926 06:57:30.206154 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "8b3706b13248460dbee8a8c74c5e3faa"
format_stamp: "Formatted at 2026-09-26 06:57:30 on dist-test-slave-sfvt"
server_key: "df59d0d7dd15944bb600f5a4424172c7"
server_key_iv: "0e38544cd20d92fb859bd7a1d0d5ed53"
server_key_version: "encryptionkey@0"
I20260926 06:57:30.207481 4571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:57:30.208204 4586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:30.208380 4571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:30.208498 4571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "8b3706b13248460dbee8a8c74c5e3faa"
format_stamp: "Formatted at 2026-09-26 06:57:30 on dist-test-slave-sfvt"
server_key: "df59d0d7dd15944bb600f5a4424172c7"
server_key_iv: "0e38544cd20d92fb859bd7a1d0d5ed53"
server_key_version: "encryptionkey@0"
I20260926 06:57:30.208585 4571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:30.245329 4571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:30.247784 4571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:30.247948 4571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:30.248186 4571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:30.248507 4571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:30.248559 4571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:30.248601 4571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:30.248629 4571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:30.254623 4571 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:39789
I20260926 06:57:30.254673 4699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:39789 every 8 connection(s)
I20260926 06:57:30.254972 4571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:30.263145 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 4571
I20260926 06:57:30.263235 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:30.263449 1321 external_mini_cluster.cc:1428] Setting key f573fafdf73fbe619c2adf8e686b58ed
I20260926 06:57:30.263731 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:30.255653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:55623 (local address 127.1.74.126:36707)
0926 06:57:30.255728 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:30.255731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:30.255939 (+ 208us) server_negotiation.cc:407] Connection header received
0926 06:57:30.256391 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:30.256393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:30.256404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:30.256457 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:30.257336 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.257896 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.258883 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.259069 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.259121 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:30.261219 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:30.261221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:30.261221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:30.261227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:30.262385 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:30.262723 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:30.262724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:30.262724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:30.262770 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:30.262997 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:30.262997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:30.262997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:30.263089 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:30.263138 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:30.263504 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 06:57:30.263616 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:57:30.263861 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:30.255636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36707 (local address 127.1.74.66:55623)
0926 06:57:30.255823 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:30.255858 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:30.256315 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:30.256525 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:30.256528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:30.257078 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:30.257266 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:30.257271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.257967 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.257968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:30.258779 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:30.258781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.259136 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.259137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:30.259241 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:30.259811 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:30.259833 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:30.261140 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:30.262452 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:30.262457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:30.262459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:30.262660 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:30.262833 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:30.262834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:30.262835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:30.262925 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:30.263149 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:30.263155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:30.263351 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 06:57:30.263472 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 06:57:30.263631 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
I20260926 06:57:30.264423 4700 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36707
I20260926 06:57:30.264518 4700 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:30.264703 4700 heartbeater.cc:511] Master 127.1.74.126:36707 requested a full tablet report, sending...
I20260926 06:57:30.265043 4375 ts_manager.cc:194] Registered new tserver with Master: 8b3706b13248460dbee8a8c74c5e3faa (127.1.74.66:39789)
I20260926 06:57:30.265430 4375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:55623
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:30.286242 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36707
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:30.368058 4705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:30.368264 4705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:30.368288 4705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:30.369377 4705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:30.369413 4705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:30.369439 4705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:30.370795 4705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:30.370990 4705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:30.371158 4705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:30.376603 4713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:30.376605 4710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:30.376600 4711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:30.376842 4705 server_base.cc:1034] running on GCE node
I20260926 06:57:30.377084 4705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:30.377326 4705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:30.378458 4705 hybrid_clock.cc:648] HybridClock initialized: now 1790405850378442 us; error 33 us; skew 500 ppm
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:30.380853 4705 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:30.381253 4705 webserver.cc:460] Webserver started at http://127.1.74.67:36957/ using document root <none> and password file <none>
I20260926 06:57:30.381492 4705 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:30.381558 4705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:30.381705 4705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:30.382732 4705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "888c9b145636452796a521327b015a90"
format_stamp: "Formatted at 2026-09-26 06:57:30 on dist-test-slave-sfvt"
server_key: "20a1a25f5f1a6d90c3fae39e21be41c4"
server_key_iv: "9a674913a9a75f3c6a27e1f171a3b4dc"
server_key_version: "encryptionkey@0"
I20260926 06:57:30.383093 4705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "888c9b145636452796a521327b015a90"
format_stamp: "Formatted at 2026-09-26 06:57:30 on dist-test-slave-sfvt"
server_key: "20a1a25f5f1a6d90c3fae39e21be41c4"
server_key_iv: "9a674913a9a75f3c6a27e1f171a3b4dc"
server_key_version: "encryptionkey@0"
I20260926 06:57:30.384473 4705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:30.385185 4720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:30.385389 4705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:30.385483 4705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "888c9b145636452796a521327b015a90"
format_stamp: "Formatted at 2026-09-26 06:57:30 on dist-test-slave-sfvt"
server_key: "20a1a25f5f1a6d90c3fae39e21be41c4"
server_key_iv: "9a674913a9a75f3c6a27e1f171a3b4dc"
server_key_version: "encryptionkey@0"
I20260926 06:57:30.385581 4705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:30.399978 4705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:30.402024 4705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:30.402186 4705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:30.402398 4705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:30.402683 4705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:30.402733 4705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:30.402767 4705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:30.402796 4705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:30.408465 4705 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:45003
I20260926 06:57:30.408557 4833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:45003 every 8 connection(s)
I20260926 06:57:30.408841 4705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:57:30.410729 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 4705
I20260926 06:57:30.410835 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:30.410991 1321 external_mini_cluster.cc:1428] Setting key 0a8b8875753047bae9d0c9b40b946bee
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405850, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:30.418329 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:30.409651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:55275 (local address 127.1.74.126:36707)
0926 06:57:30.409714 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:30.409716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:30.409924 (+ 208us) server_negotiation.cc:407] Connection header received
0926 06:57:30.410387 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:30.410388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:30.410397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:30.410463 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:30.411587 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.412089 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.413155 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.413367 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.413459 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:30.415578 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:30.415580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:30.415580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:30.415588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:30.416964 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:30.417380 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:30.417381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:30.417381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:30.417442 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:30.417706 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:30.417706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:30.417707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:30.417821 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:30.417889 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:30.418164 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 06:57:30.418239 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:57:30.418517 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:30.409632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36707 (local address 127.1.74.67:55275)
0926 06:57:30.409800 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:30.409841 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:30.410305 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:30.410516 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:30.410519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:30.411241 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:30.411494 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:30.411498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.412233 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.412234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:30.413063 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:30.413065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.413468 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.413469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:30.413609 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:30.414133 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:30.414141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:30.415487 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:30.417065 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:30.417071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:30.417073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:30.417290 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:30.417508 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:30.417509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:30.417509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:30.417629 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:30.417892 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:30.417898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:30.418031 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 06:57:30.418142 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 06:57:30.418291 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260926 06:57:30.418941 4834 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36707
I20260926 06:57:30.419023 4834 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:30.419243 4834 heartbeater.cc:511] Master 127.1.74.126:36707 requested a full tablet report, sending...
I20260926 06:57:30.419665 4375 ts_manager.cc:194] Registered new tserver with Master: 888c9b145636452796a521327b015a90 (127.1.74.67:45003)
I20260926 06:57:30.420040 4375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:55275
I20260926 06:57:30.422446 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405850, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405850, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:30.444994 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:30.437885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.1.74.126:36707)
0926 06:57:30.437949 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:30.437951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:30.437991 (+ 40us) server_negotiation.cc:407] Connection header received
0926 06:57:30.438067 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:30.438068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:30.438078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:30.438136 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:30.438682 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.439163 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.439926 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:30.440103 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:30.440154 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:30.442572 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:30.442573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:30.442573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:30.442580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:30.443893 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:30.444218 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:30.444219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:30.444219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:30.444305 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:30.444517 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:30.444518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:30.444518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:30.444695 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:30.444784 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:30.444865 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 06:57:30.444928 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 06:57:30.445262 4375 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46194
I20260926 06:57:30.445714 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 4437
I20260926 06:57:30.450068 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 4571
I20260926 06:57:30.454339 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 4705
I20260926 06:57:30.458534 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 4343
2026-09-26T06:57:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1879 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4846](info): set up 2 sockets
Sep 26 06:57:30 dist-test-slave-sfvt krb5kdc[4846](info): commencing operation
krb5kdc: starting...
W20260926 06:57:31.507393 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:31 dist-test-slave-sfvt krb5kdc[4846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:31.625979 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:34671
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:34671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:31.706959 4859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:31.707159 4859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:31.707180 4859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:31.708343 4859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:31.708384 4859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:31.708397 4859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:31.708410 4859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:31.708421 4859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:31.709815 4859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:34671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:34671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:31.710003 4859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:31.710161 4859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:31.715291 4865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:31.715341 4864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:31.715291 4867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:31.715313 4859 server_base.cc:1034] running on GCE node
I20260926 06:57:31.715727 4859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:31.716002 4859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:31.717172 4859 hybrid_clock.cc:648] HybridClock initialized: now 1790405851717175 us; error 49 us; skew 500 ppm
Sep 26 06:57:31 dist-test-slave-sfvt krb5kdc[4846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:31.719676 4859 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:31.720124 4859 webserver.cc:460] Webserver started at http://127.1.74.126:41395/ using document root <none> and password file <none>
I20260926 06:57:31.720300 4859 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:31.720348 4859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:31.720432 4859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:31.721410 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "a3a52431a2464696ae94af4dae2020d0"
format_stamp: "Formatted at 2026-09-26 06:57:31 on dist-test-slave-sfvt"
server_key: "5a240c0691869032e1167f1fa552c36e"
server_key_iv: "74474e2c2f71c9930320a954ada56a71"
server_key_version: "encryptionkey@0"
I20260926 06:57:31.721696 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "a3a52431a2464696ae94af4dae2020d0"
format_stamp: "Formatted at 2026-09-26 06:57:31 on dist-test-slave-sfvt"
server_key: "5a240c0691869032e1167f1fa552c36e"
server_key_iv: "74474e2c2f71c9930320a954ada56a71"
server_key_version: "encryptionkey@0"
I20260926 06:57:31.722942 4859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 06:57:31.723673 4874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:31.723846 4859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:31.723942 4859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "a3a52431a2464696ae94af4dae2020d0"
format_stamp: "Formatted at 2026-09-26 06:57:31 on dist-test-slave-sfvt"
server_key: "5a240c0691869032e1167f1fa552c36e"
server_key_iv: "74474e2c2f71c9930320a954ada56a71"
server_key_version: "encryptionkey@0"
I20260926 06:57:31.724022 4859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:31.734666 4859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:31.736430 4859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:31.736584 4859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:31.740450 4859 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:34671
I20260926 06:57:31.740481 4926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:34671 every 8 connection(s)
I20260926 06:57:31.740950 4859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:31.741561 4927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:31.743757 4927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0: Bootstrap starting.
I20260926 06:57:31.744261 4927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:31.744498 4927 log.cc:824] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:31.745128 4927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0: No bootstrap required, opened a new log
I20260926 06:57:31.746595 4927 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a52431a2464696ae94af4dae2020d0" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34671 } }
I20260926 06:57:31.746724 4927 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:31.746747 4927 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a52431a2464696ae94af4dae2020d0, State: Initialized, Role: FOLLOWER
I20260926 06:57:31.746876 4927 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a52431a2464696ae94af4dae2020d0" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34671 } }
I20260926 06:57:31.746953 4927 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:31.746975 4927 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:31.747004 4927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:31.747857 4927 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a52431a2464696ae94af4dae2020d0" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34671 } }
I20260926 06:57:31.747941 4927 leader_election.cc:302] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3a52431a2464696ae94af4dae2020d0; no voters:
I20260926 06:57:31.748059 4927 leader_election.cc:288] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:31.748136 4930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:31.748301 4927 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:31.748376 4930 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [term 1 LEADER]: Becoming Leader. State: Replica: a3a52431a2464696ae94af4dae2020d0, State: Running, Role: LEADER
I20260926 06:57:31.748464 4930 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a52431a2464696ae94af4dae2020d0" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34671 } }
I20260926 06:57:31.748847 4932 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3a52431a2464696ae94af4dae2020d0. Latest consensus state: current_term: 1 leader_uuid: "a3a52431a2464696ae94af4dae2020d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a52431a2464696ae94af4dae2020d0" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34671 } } }
I20260926 06:57:31.748847 4931 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3a52431a2464696ae94af4dae2020d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a52431a2464696ae94af4dae2020d0" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 34671 } } }
I20260926 06:57:31.748955 4932 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:31.748977 4931 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:31.749148 4939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:31.749699 4939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:31.750898 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 4859
I20260926 06:57:31.751019 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:31.751168 4939 catalog_manager.cc:1347] Generated new cluster ID: 433e59ee25ee4d41b318a67daace2f84
I20260926 06:57:31.751222 4939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:31.751201 1321 external_mini_cluster.cc:1428] Setting key 700e262cbbacba18cb3c55358f78e944
Sep 26 06:57:31 dist-test-slave-sfvt krb5kdc[4846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:31.760648 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:31.751698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.1.74.126:34671)
0926 06:57:31.751847 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:31.751854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:31.752018 (+ 164us) server_negotiation.cc:407] Connection header received
0926 06:57:31.752171 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:31.752176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:31.752234 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:31.752366 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:31.752966 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:31.753893 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:31.754651 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:31.755076 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:31.755193 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:31.757414 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:31.757417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:31.757418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:31.757431 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:31.759041 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:31.759483 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:31.759486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:31.759487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:31.759576 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:31.759784 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:31.759785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:31.759785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:31.760033 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:31.760228 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:31.760295 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 06:57:31.760434 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260926 06:57:31.768025 4939 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:31.768527 4939 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:31.774478 4939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3a52431a2464696ae94af4dae2020d0: Generated new TSK 0
I20260926 06:57:31.774655 4939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:31.800503 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34671
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:31.882143 4953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:31.882368 4953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:31.882395 4953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:31.883598 4953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:31.883644 4953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:31.883705 4953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:31.885221 4953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:31.885490 4953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:31.885735 4953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:31.890750 4961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:31.890851 4959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:31.890974 4953 server_base.cc:1034] running on GCE node
W20260926 06:57:31.890777 4958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:31.891317 4953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:31.891534 4953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:31.892714 4953 hybrid_clock.cc:648] HybridClock initialized: now 1790405851892702 us; error 39 us; skew 500 ppm
Sep 26 06:57:31 dist-test-slave-sfvt krb5kdc[4846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:31.895332 4953 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:31.895756 4953 webserver.cc:460] Webserver started at http://127.1.74.65:38413/ using document root <none> and password file <none>
I20260926 06:57:31.895993 4953 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:31.896039 4953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:31.896180 4953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:31.897230 4953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "8ccc49df8f174f6f928d7fac1d609608"
format_stamp: "Formatted at 2026-09-26 06:57:31 on dist-test-slave-sfvt"
server_key: "3f58525500d10b1096cd87c294512ffd"
server_key_iv: "442c8a5e2bd7dc85e1264f5e79ba1918"
server_key_version: "encryptionkey@0"
I20260926 06:57:31.897601 4953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "8ccc49df8f174f6f928d7fac1d609608"
format_stamp: "Formatted at 2026-09-26 06:57:31 on dist-test-slave-sfvt"
server_key: "3f58525500d10b1096cd87c294512ffd"
server_key_iv: "442c8a5e2bd7dc85e1264f5e79ba1918"
server_key_version: "encryptionkey@0"
I20260926 06:57:31.898942 4953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:57:31.899727 4968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:31.899874 4953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:31.899962 4953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "8ccc49df8f174f6f928d7fac1d609608"
format_stamp: "Formatted at 2026-09-26 06:57:31 on dist-test-slave-sfvt"
server_key: "3f58525500d10b1096cd87c294512ffd"
server_key_iv: "442c8a5e2bd7dc85e1264f5e79ba1918"
server_key_version: "encryptionkey@0"
I20260926 06:57:31.900030 4953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:31.920210 4953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:31.922603 4953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:31.922714 4953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:31.922899 4953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:31.923213 4953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:31.923246 4953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:31.923293 4953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:31.923307 4953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:31.929111 4953 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:35985
I20260926 06:57:31.929170 5081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:35985 every 8 connection(s)
I20260926 06:57:31.929509 4953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:31.935628 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 4953
I20260926 06:57:31.935736 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
Sep 26 06:57:31 dist-test-slave-sfvt krb5kdc[4846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405851, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:31.935928 1321 external_mini_cluster.cc:1428] Setting key 1572787f2afb213abce7ade8be7b05d7
I20260926 06:57:31.939294 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:31.930235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:59271 (local address 127.1.74.126:34671)
0926 06:57:31.930282 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:31.930285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:31.930476 (+ 191us) server_negotiation.cc:407] Connection header received
0926 06:57:31.931539 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:31.931540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:31.931554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:31.931604 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:31.932217 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:31.932820 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:31.933811 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:31.934026 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:31.934100 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:31.936394 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:31.936395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:31.936396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:31.936414 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:31.937823 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:31.938271 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:31.938272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:31.938273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:31.938330 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:31.938607 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:31.938608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:31.938609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:31.938727 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:31.938806 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:31.939087 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 06:57:31.939170 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:57:31.939453 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:31.930199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:34671 (local address 127.1.74.65:59271)
0926 06:57:31.930330 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:31.930378 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:31.931424 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:31.931673 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:31.931677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:31.931927 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:31.932109 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:31.932113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:31.932926 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:31.932927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:31.933720 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:31.933722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:31.934116 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:31.934117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:31.934255 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:31.934687 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:31.934694 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:31.936287 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:31.937947 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:31.937953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:31.937956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:31.938177 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:31.938397 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:31.938397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:31.938398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:31.938535 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:31.938795 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:31.938801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:31.938943 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 06:57:31.939052 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 06:57:31.939204 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260926 06:57:31.939975 5082 heartbeater.cc:347] Connected to a master server at 127.1.74.126:34671
I20260926 06:57:31.940068 5082 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:31.940253 5082 heartbeater.cc:511] Master 127.1.74.126:34671 requested a full tablet report, sending...
I20260926 06:57:31.940761 4891 ts_manager.cc:194] Registered new tserver with Master: 8ccc49df8f174f6f928d7fac1d609608 (127.1.74.65:35985)
I20260926 06:57:31.941360 4891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:59271
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:31.963042 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34671
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:32.044600 5087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:32.044816 5087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:32.044840 5087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:32.045903 5087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:32.045938 5087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:32.045964 5087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:32.047458 5087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:32.047638 5087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:32.047816 5087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:57:32.052947 5087 server_base.cc:1034] running on GCE node
W20260926 06:57:32.052968 5092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:32.052918 5093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:32.052913 5095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:32.053434 5087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:32.053670 5087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:32.054852 5087 hybrid_clock.cc:648] HybridClock initialized: now 1790405852054825 us; error 39 us; skew 500 ppm
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[4846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:32.057327 5087 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:32.057845 5087 webserver.cc:460] Webserver started at http://127.1.74.66:40829/ using document root <none> and password file <none>
I20260926 06:57:32.058028 5087 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:32.058068 5087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:32.058156 5087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:32.059114 5087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "000761ae451e4049ad46c2b1598627d6"
format_stamp: "Formatted at 2026-09-26 06:57:32 on dist-test-slave-sfvt"
server_key: "18d9d377351b221393d2340e6f3159ed"
server_key_iv: "3b110a18cc0388f09d2cfd3d10188b88"
server_key_version: "encryptionkey@0"
I20260926 06:57:32.059473 5087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "000761ae451e4049ad46c2b1598627d6"
format_stamp: "Formatted at 2026-09-26 06:57:32 on dist-test-slave-sfvt"
server_key: "18d9d377351b221393d2340e6f3159ed"
server_key_iv: "3b110a18cc0388f09d2cfd3d10188b88"
server_key_version: "encryptionkey@0"
I20260926 06:57:32.060745 5087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:32.061529 5102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:32.061697 5087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:32.061776 5087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "000761ae451e4049ad46c2b1598627d6"
format_stamp: "Formatted at 2026-09-26 06:57:32 on dist-test-slave-sfvt"
server_key: "18d9d377351b221393d2340e6f3159ed"
server_key_iv: "3b110a18cc0388f09d2cfd3d10188b88"
server_key_version: "encryptionkey@0"
I20260926 06:57:32.061826 5087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:32.080506 5087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:32.082724 5087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:32.082855 5087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:32.083029 5087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:32.083359 5087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:32.083392 5087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:32.083413 5087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:32.083461 5087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:32.088960 5087 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:40469
I20260926 06:57:32.089040 5215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:40469 every 8 connection(s)
I20260926 06:57:32.089330 5087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[4846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:32.098078 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 5087
I20260926 06:57:32.098198 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:32.098392 1321 external_mini_cluster.cc:1428] Setting key 32f3f95d1f310839b9f81e24451b73c7
I20260926 06:57:32.098568 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:32.090069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:35777 (local address 127.1.74.126:34671)
0926 06:57:32.090129 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:32.090131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:32.090336 (+ 205us) server_negotiation.cc:407] Connection header received
0926 06:57:32.091227 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:32.091228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:32.091237 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:32.091353 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:32.092067 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.092550 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.093509 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.093690 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.093739 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:32.095881 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:32.095882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:32.095883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:32.095890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:32.097144 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:32.097584 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:32.097585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:32.097585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:32.097647 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:32.097900 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:32.097901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:32.097901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:32.098017 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:32.098091 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:32.098382 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 06:57:32.098478 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:57:32.098757 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:32.090032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:34671 (local address 127.1.74.66:35777)
0926 06:57:32.090201 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:32.090242 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:32.091144 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:32.091525 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:32.091529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:32.091798 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:32.091986 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:32.091990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.092631 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.092632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:32.093436 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:32.093438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.093759 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.093760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:32.093854 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:32.094286 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:32.094294 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:32.095765 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:32.097247 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:32.097253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:32.097256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:32.097493 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:32.097719 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:32.097720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:32.097720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:32.097824 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:32.098081 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:32.098087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:32.098248 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:57:32.098362 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 06:57:32.098520 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260926 06:57:32.099298 5216 heartbeater.cc:347] Connected to a master server at 127.1.74.126:34671
I20260926 06:57:32.099398 5216 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:32.099574 5216 heartbeater.cc:511] Master 127.1.74.126:34671 requested a full tablet report, sending...
I20260926 06:57:32.099975 4891 ts_manager.cc:194] Registered new tserver with Master: 000761ae451e4049ad46c2b1598627d6 (127.1.74.66:40469)
I20260926 06:57:32.100402 4891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:35777
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:32.122593 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34671
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:32.208069 5221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:32.208312 5221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:32.208352 5221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:32.209540 5221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:32.209609 5221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:32.209653 5221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:32.211148 5221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:34671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:32.211436 5221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:32.211668 5221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:32.217103 5226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:32.217083 5227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:32.217132 5221 server_base.cc:1034] running on GCE node
W20260926 06:57:32.217105 5229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:32.217650 5221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:32.217847 5221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:32.218991 5221 hybrid_clock.cc:648] HybridClock initialized: now 1790405852218976 us; error 28 us; skew 500 ppm
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[4846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:32.221621 5221 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:32.222095 5221 webserver.cc:460] Webserver started at http://127.1.74.67:35509/ using document root <none> and password file <none>
I20260926 06:57:32.222319 5221 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:32.222364 5221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:32.222513 5221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:32.223620 5221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "f93c2a224f594918ae893f42fbb94566"
format_stamp: "Formatted at 2026-09-26 06:57:32 on dist-test-slave-sfvt"
server_key: "873383da7defdbe0364e50507b371d4c"
server_key_iv: "9a82dd2f7268b9f5bc18373b3dbe3cef"
server_key_version: "encryptionkey@0"
I20260926 06:57:32.223974 5221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "f93c2a224f594918ae893f42fbb94566"
format_stamp: "Formatted at 2026-09-26 06:57:32 on dist-test-slave-sfvt"
server_key: "873383da7defdbe0364e50507b371d4c"
server_key_iv: "9a82dd2f7268b9f5bc18373b3dbe3cef"
server_key_version: "encryptionkey@0"
I20260926 06:57:32.225317 5221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:57:32.226047 5236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:32.226218 5221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:32.226317 5221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "f93c2a224f594918ae893f42fbb94566"
format_stamp: "Formatted at 2026-09-26 06:57:32 on dist-test-slave-sfvt"
server_key: "873383da7defdbe0364e50507b371d4c"
server_key_iv: "9a82dd2f7268b9f5bc18373b3dbe3cef"
server_key_version: "encryptionkey@0"
I20260926 06:57:32.226380 5221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:32.239240 5221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:32.241149 5221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:32.241310 5221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:32.241541 5221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:32.241848 5221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:32.241902 5221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:32.241943 5221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:32.241971 5221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:32.247694 5221 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:40885
I20260926 06:57:32.247730 5349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:40885 every 8 connection(s)
I20260926 06:57:32.248148 5221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[4846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:32.257151 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:32.248917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:49719 (local address 127.1.74.126:34671)
0926 06:57:32.249010 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:32.249013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:32.249373 (+ 360us) server_negotiation.cc:407] Connection header received
0926 06:57:32.250387 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:32.250388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:32.250398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:32.250450 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:32.251004 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.251565 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.252508 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.252688 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.252736 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:32.254681 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:32.254682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:32.254683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:32.254689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:32.255961 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:32.256352 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:32.256353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:32.256353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:32.256398 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:32.256646 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:32.256647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:32.256647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:32.256743 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:32.256791 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:32.257032 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 06:57:32.257088 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 06:57:32.257387 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:32.248923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:34671 (local address 127.1.74.67:49719)
0926 06:57:32.249101 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:32.249136 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:32.250296 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:32.250504 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:32.250507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:32.250749 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:32.250938 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:32.250942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.251637 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.251637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:32.252437 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:32.252439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.252758 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.252758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:32.252853 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:32.253295 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:32.253303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:32.254601 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:32.256052 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:32.256058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:32.256060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:32.256281 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:32.256459 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:32.256459 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:32.256460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:32.256589 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:32.256798 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:32.256802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:32.256924 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 06:57:32.257007 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:57:32.257125 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260926 06:57:32.257874 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 5221
I20260926 06:57:32.257877 5350 heartbeater.cc:347] Connected to a master server at 127.1.74.126:34671
I20260926 06:57:32.257990 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:32.257989 5350 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:32.258173 5350 heartbeater.cc:511] Master 127.1.74.126:34671 requested a full tablet report, sending...
I20260926 06:57:32.258184 1321 external_mini_cluster.cc:1428] Setting key ad19a9f057c5f1ca1c647a7a511d3766
I20260926 06:57:32.258586 4890 ts_manager.cc:194] Registered new tserver with Master: f93c2a224f594918ae893f42fbb94566 (127.1.74.67:40885)
I20260926 06:57:32.258955 4890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:49719
I20260926 06:57:32.259656 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[4846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[4846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:32.282292 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:32.274456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.1.74.65:35985)
0926 06:57:32.274613 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:32.274619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:32.274631 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:57:32.274666 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:32.274669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:32.274683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:32.274748 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:32.275434 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.276150 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.276885 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.277204 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.277256 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:32.279657 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:32.279665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:32.279666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:32.279672 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:32.281266 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:32.281586 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:32.281589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:32.281589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:32.281654 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:32.281857 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:32.281857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:32.281858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:32.281980 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:32.282090 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:32.282154 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 06:57:32.282215 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":104,"threads_started":1}
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[4846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:32.290201 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:32.283597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.1.74.126:34671)
0926 06:57:32.283640 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:32.283641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:32.283656 (+ 15us) server_negotiation.cc:407] Connection header received
0926 06:57:32.283706 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:32.283706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:32.283714 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:32.283758 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:32.284215 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.284744 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.285425 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:32.285600 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:32.285654 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:32.287845 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:32.287847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:32.287847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:32.287855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:32.288999 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:32.289325 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:32.289327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:32.289327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:32.289423 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:32.289629 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:32.289630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:32.289630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:32.289774 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:32.289872 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:32.290027 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 06:57:32.290101 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 06:57:32.290465 4890 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33736
I20260926 06:57:32.290813 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 4953
I20260926 06:57:32.295761 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 5087
I20260926 06:57:32.299794 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 5221
I20260926 06:57:32.303807 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 4859
2026-09-26T06:57:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1845 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[5362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[5362](info): set up 2 sockets
Sep 26 06:57:32 dist-test-slave-sfvt krb5kdc[5362](info): commencing operation
krb5kdc: starting...
W20260926 06:57:33.347316 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:33 dist-test-slave-sfvt krb5kdc[5362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:33.466961 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45265
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:45265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:33.546627 5375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:33.546878 5375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:33.546914 5375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:33.548116 5375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:33.548202 5375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:33.548226 5375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:33.548257 5375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:33.548285 5375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:33.549804 5375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:45265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:33.550084 5375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:33.550311 5375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:33.554960 5383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:33.554983 5380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:33.554997 5381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:33.555223 5375 server_base.cc:1034] running on GCE node
I20260926 06:57:33.555470 5375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:33.555770 5375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:33.556969 5375 hybrid_clock.cc:648] HybridClock initialized: now 1790405853556956 us; error 29 us; skew 500 ppm
Sep 26 06:57:33 dist-test-slave-sfvt krb5kdc[5362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:33.559542 5375 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:33.559909 5375 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:33.559957 5375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:33.560247 5375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:33.561460 5375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "6f6efa1b5b724dd2aa212986168cd19d"
format_stamp: "Formatted at 2026-09-26 06:57:33 on dist-test-slave-sfvt"
server_key: "f5c35bc35ed1df708ab229926b5a95e9"
server_key_iv: "e9df5a99ea2da5bb06c727c4ca120fe1"
server_key_version: "encryptionkey@0"
I20260926 06:57:33.561822 5375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "6f6efa1b5b724dd2aa212986168cd19d"
format_stamp: "Formatted at 2026-09-26 06:57:33 on dist-test-slave-sfvt"
server_key: "f5c35bc35ed1df708ab229926b5a95e9"
server_key_iv: "e9df5a99ea2da5bb06c727c4ca120fe1"
server_key_version: "encryptionkey@0"
I20260926 06:57:33.563071 5375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:57:33.563860 5389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:33.564029 5375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:33.564112 5375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "6f6efa1b5b724dd2aa212986168cd19d"
format_stamp: "Formatted at 2026-09-26 06:57:33 on dist-test-slave-sfvt"
server_key: "f5c35bc35ed1df708ab229926b5a95e9"
server_key_iv: "e9df5a99ea2da5bb06c727c4ca120fe1"
server_key_version: "encryptionkey@0"
I20260926 06:57:33.564165 5375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:33.571133 5375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:33.573107 5375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:33.573226 5375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:33.576896 5375 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:45265
I20260926 06:57:33.576920 5441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:45265 every 8 connection(s)
I20260926 06:57:33.577229 5375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:33.578195 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:33.580549 5442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d: Bootstrap starting.
I20260926 06:57:33.581123 5442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:33.581477 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 5375
I20260926 06:57:33.581506 5442 log.cc:824] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:33.581576 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:33.581755 1321 external_mini_cluster.cc:1428] Setting key dfe971e974fbf55aa09803b84170bfc3
I20260926 06:57:33.582566 5442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d: No bootstrap required, opened a new log
I20260926 06:57:33.584563 5442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6efa1b5b724dd2aa212986168cd19d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45265 } }
I20260926 06:57:33.584714 5442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:33.584776 5442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6efa1b5b724dd2aa212986168cd19d, State: Initialized, Role: FOLLOWER
I20260926 06:57:33.584924 5442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6efa1b5b724dd2aa212986168cd19d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45265 } }
I20260926 06:57:33.584991 5442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:33.585012 5442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:33.585078 5442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:33.585616 5442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6efa1b5b724dd2aa212986168cd19d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45265 } }
I20260926 06:57:33.585734 5442 leader_election.cc:302] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f6efa1b5b724dd2aa212986168cd19d; no voters:
I20260926 06:57:33.585934 5442 leader_election.cc:288] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:33.585952 5447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:33.586077 5447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [term 1 LEADER]: Becoming Leader. State: Replica: 6f6efa1b5b724dd2aa212986168cd19d, State: Running, Role: LEADER
I20260926 06:57:33.586155 5447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6efa1b5b724dd2aa212986168cd19d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45265 } }
I20260926 06:57:33.586210 5442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:33.586510 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f6efa1b5b724dd2aa212986168cd19d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6efa1b5b724dd2aa212986168cd19d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45265 } } }
I20260926 06:57:33.586934 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:33.587103 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f6efa1b5b724dd2aa212986168cd19d. Latest consensus state: current_term: 1 leader_uuid: "6f6efa1b5b724dd2aa212986168cd19d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6efa1b5b724dd2aa212986168cd19d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45265 } } }
I20260926 06:57:33.587164 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:33.587450 5456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:33.587994 5456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 06:57:33 dist-test-slave-sfvt krb5kdc[5362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:33.589598 5456 catalog_manager.cc:1347] Generated new cluster ID: 96fcf522ebf0408abd297f4f4397603e
I20260926 06:57:33.589645 5456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:33.595161 5456 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:33.595888 5456 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:33.597410 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:33.582266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.1.74.126:45265)
0926 06:57:33.582432 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:33.582443 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:33.582482 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:57:33.582937 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:33.582945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:33.582991 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:33.583136 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:33.583786 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.584635 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.585553 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.586140 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.586267 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:33.589175 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:33.589179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:33.589179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:33.589204 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:33.595843 (+ 6639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:33.596260 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:33.596263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:33.596263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:33.596353 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:33.596571 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:33.596572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:33.596572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:33.596783 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:33.596957 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:33.597052 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 06:57:33.597211 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260926 06:57:33.598775 5456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d: Generated new TSK 0
I20260926 06:57:33.598950 5456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:33.628166 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45265
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:33.710177 5468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:33.710402 5468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:33.710425 5468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:33.711622 5468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:33.711668 5468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:33.711695 5468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:33.713290 5468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:33.713549 5468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:33.713797 5468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:33.718616 5476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:33.718659 5473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:33.718700 5474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:33.718806 5468 server_base.cc:1034] running on GCE node
I20260926 06:57:33.719054 5468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:33.719393 5468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:33.720556 5468 hybrid_clock.cc:648] HybridClock initialized: now 1790405853720539 us; error 44 us; skew 500 ppm
Sep 26 06:57:33 dist-test-slave-sfvt krb5kdc[5362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:33.723022 5468 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:33.723453 5468 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:33.723531 5468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:33.723690 5468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:33.724743 5468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "afe44a70684b4056a154a51ff341333b"
format_stamp: "Formatted at 2026-09-26 06:57:33 on dist-test-slave-sfvt"
server_key: "7ad13ca3c1bb90a338f59aff42d64e6c"
server_key_iv: "a1d4960665ad419757ba7780acfbcf95"
server_key_version: "encryptionkey@0"
I20260926 06:57:33.725114 5468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "afe44a70684b4056a154a51ff341333b"
format_stamp: "Formatted at 2026-09-26 06:57:33 on dist-test-slave-sfvt"
server_key: "7ad13ca3c1bb90a338f59aff42d64e6c"
server_key_iv: "a1d4960665ad419757ba7780acfbcf95"
server_key_version: "encryptionkey@0"
I20260926 06:57:33.726384 5468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:33.727160 5482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:33.727353 5468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:33.727450 5468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "afe44a70684b4056a154a51ff341333b"
format_stamp: "Formatted at 2026-09-26 06:57:33 on dist-test-slave-sfvt"
server_key: "7ad13ca3c1bb90a338f59aff42d64e6c"
server_key_iv: "a1d4960665ad419757ba7780acfbcf95"
server_key_version: "encryptionkey@0"
I20260926 06:57:33.727537 5468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:33.758984 5468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:33.761111 5468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:33.761289 5468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:33.761529 5468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:33.761850 5468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:33.761905 5468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:33.761947 5468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:33.761976 5468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:33.768167 5468 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:39279
I20260926 06:57:33.768241 5595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:39279 every 8 connection(s)
I20260926 06:57:33.768540 5468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:33.773147 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 5468
I20260926 06:57:33.773229 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:33.773417 1321 external_mini_cluster.cc:1428] Setting key 50fb1689eb91ba8912dfb0d568fc6446
Sep 26 06:57:33 dist-test-slave-sfvt krb5kdc[5362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:33.779170 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:33.769229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45265 (local address 127.1.74.65:47367)
0926 06:57:33.769371 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:33.769417 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:33.769929 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:33.770193 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:33.770197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:33.770463 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:33.770962 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:33.770966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.771778 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.771779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:33.772578 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:33.772580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.772999 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.773000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:33.773128 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:33.773682 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:33.773698 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:33.775573 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:33.777502 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:33.777507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:33.777509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:33.777967 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:33.778219 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:33.778220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:33.778220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:33.778342 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:33.778646 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:33.778651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:33.778770 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 06:57:33.778871 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 06:57:33.779008 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260926 06:57:33.779168 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:33.769263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:47367 (local address 127.1.74.126:45265)
0926 06:57:33.769303 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:33.769305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:33.769509 (+ 204us) server_negotiation.cc:407] Connection header received
0926 06:57:33.770044 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:33.770045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:33.770058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:33.770110 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:33.771067 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.771668 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.772685 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.772903 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.772979 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:33.775709 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:33.775711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:33.775711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:33.775721 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:33.777403 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:33.778073 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:33.778074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:33.778075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:33.778144 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:33.778415 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:33.778416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:33.778417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:33.778556 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:33.778631 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:33.778930 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 06:57:33.779033 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:57:33.779760 5596 heartbeater.cc:347] Connected to a master server at 127.1.74.126:45265
I20260926 06:57:33.779882 5596 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:33.780046 5596 heartbeater.cc:511] Master 127.1.74.126:45265 requested a full tablet report, sending...
I20260926 06:57:33.780627 5406 ts_manager.cc:194] Registered new tserver with Master: afe44a70684b4056a154a51ff341333b (127.1.74.65:39279)
I20260926 06:57:33.781591 5406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:47367
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:33.800069 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45265
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:33.880383 5601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:33.880592 5601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:33.880617 5601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:33.881698 5601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:33.881732 5601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:33.881759 5601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:33.883230 5601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:33.883432 5601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:33.883610 5601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:33.888645 5609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:33.888645 5606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:33.888744 5607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:33.888974 5601 server_base.cc:1034] running on GCE node
I20260926 06:57:33.889207 5601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:33.889457 5601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:33.890597 5601 hybrid_clock.cc:648] HybridClock initialized: now 1790405853890589 us; error 33 us; skew 500 ppm
Sep 26 06:57:33 dist-test-slave-sfvt krb5kdc[5362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:33.893270 5601 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:33.893743 5601 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:33.893826 5601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:33.893962 5601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:33.895116 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "8fb2df5f275045c6b055228366697308"
format_stamp: "Formatted at 2026-09-26 06:57:33 on dist-test-slave-sfvt"
server_key: "65d0f4141f3af59948a9527515947bf7"
server_key_iv: "388bd5313c57073f1051eb8e0d36386b"
server_key_version: "encryptionkey@0"
I20260926 06:57:33.895519 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "8fb2df5f275045c6b055228366697308"
format_stamp: "Formatted at 2026-09-26 06:57:33 on dist-test-slave-sfvt"
server_key: "65d0f4141f3af59948a9527515947bf7"
server_key_iv: "388bd5313c57073f1051eb8e0d36386b"
server_key_version: "encryptionkey@0"
I20260926 06:57:33.896831 5601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:33.897600 5615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:33.897814 5601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:33.897907 5601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "8fb2df5f275045c6b055228366697308"
format_stamp: "Formatted at 2026-09-26 06:57:33 on dist-test-slave-sfvt"
server_key: "65d0f4141f3af59948a9527515947bf7"
server_key_iv: "388bd5313c57073f1051eb8e0d36386b"
server_key_version: "encryptionkey@0"
I20260926 06:57:33.897984 5601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:33.915664 5601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:33.917627 5601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:33.917800 5601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:33.918035 5601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:33.918345 5601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:33.918396 5601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:33.918437 5601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:33.918464 5601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:33.924146 5601 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:42927
I20260926 06:57:33.924178 5728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:42927 every 8 connection(s)
I20260926 06:57:33.924468 5601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:33.924899 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 5601
I20260926 06:57:33.925007 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:33.925236 1321 external_mini_cluster.cc:1428] Setting key 4ffade3e3510dfb36283785f3fbe51dd
Sep 26 06:57:33 dist-test-slave-sfvt krb5kdc[5362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
I20260926 06:57:33.934906 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:33.925189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:41417 (local address 127.1.74.126:45265)
0926 06:57:33.925257 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:33.925259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:33.925483 (+ 224us) server_negotiation.cc:407] Connection header received
0926 06:57:33.926160 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:33.926161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:33.926174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:33.926252 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:33.927248 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.927796 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.929368 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.929548 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.929601 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:33.932112 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:33.932113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:33.932114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:33.932121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:33.933503 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:33.933926 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:33.933928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:33.933928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:33.933988 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:33.934254 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:33.934255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:33.934255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:33.934367 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:33.934443 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:33.934729 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 06:57:33.934809 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:57:33.935082 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:33.925184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45265 (local address 127.1.74.66:41417)
0926 06:57:33.925347 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:33.925394 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:33.926056 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:33.926307 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:33.926310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:33.926561 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:33.927129 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:33.927137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.927889 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.927890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:33.929280 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:33.929283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:33.929627 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:33.929628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:33.929772 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:33.930315 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:33.930325 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:33.932014 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:33.933599 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:33.933605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:33.933607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:33.933849 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:33.934057 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:33.934058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:33.934058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:33.934167 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:33.934434 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:33.934440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:33.934580 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 06:57:33.934689 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 06:57:33.934848 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260926 06:57:33.935583 5729 heartbeater.cc:347] Connected to a master server at 127.1.74.126:45265
I20260926 06:57:33.935662 5729 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:33.935806 5729 heartbeater.cc:511] Master 127.1.74.126:45265 requested a full tablet report, sending...
I20260926 06:57:33.936197 5406 ts_manager.cc:194] Registered new tserver with Master: 8fb2df5f275045c6b055228366697308 (127.1.74.66:42927)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
I20260926 06:57:33.936637 5406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:41417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:33.950639 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45265
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:34.031769 5734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:34.031975 5734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:34.031998 5734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:34.033133 5734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:34.033169 5734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:34.033196 5734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:34.034598 5734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:34.034814 5734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:34.035024 5734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:34.039912 5740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:34.039945 5739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:34.040076 5734 server_base.cc:1034] running on GCE node
W20260926 06:57:34.039945 5742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:34.040366 5734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:34.040601 5734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:34.041766 5734 hybrid_clock.cc:648] HybridClock initialized: now 1790405854041754 us; error 45 us; skew 500 ppm
Sep 26 06:57:34 dist-test-slave-sfvt krb5kdc[5362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:34.044343 5734 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:34.044704 5734 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:34.044772 5734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:34.044924 5734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:34.045961 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "2d75c5104f66442791bc9fc5ea16404a"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "0a38ac39631520904186d7ae5708f123"
server_key_iv: "516f8cd9afc8021bbe0b4a614bd632c3"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.046329 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "2d75c5104f66442791bc9fc5ea16404a"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "0a38ac39631520904186d7ae5708f123"
server_key_iv: "516f8cd9afc8021bbe0b4a614bd632c3"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.047672 5734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:34.048369 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.048538 5734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.048621 5734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "2d75c5104f66442791bc9fc5ea16404a"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "0a38ac39631520904186d7ae5708f123"
server_key_iv: "516f8cd9afc8021bbe0b4a614bd632c3"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.048673 5734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:34.060599 5734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:34.062433 5734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:34.062539 5734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:34.062716 5734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:34.063024 5734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:34.063058 5734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.063103 5734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:34.063130 5734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.068853 5734 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:45913
I20260926 06:57:34.068921 5861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:45913 every 8 connection(s)
I20260926 06:57:34.069192 5734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:57:34 dist-test-slave-sfvt krb5kdc[5362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:34.075474 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 5734
I20260926 06:57:34.075552 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:34.075805 1321 external_mini_cluster.cc:1428] Setting key 20128613493f0aba6bacfd847d22db09
I20260926 06:57:34.078373 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.069946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:42185 (local address 127.1.74.126:45265)
0926 06:57:34.070013 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.070015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.070203 (+ 188us) server_negotiation.cc:407] Connection header received
0926 06:57:34.070745 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.070747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.070760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.070815 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.071809 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.072311 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.073341 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.073537 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.073615 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.075692 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.075694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.075695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:34.075702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.077246 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:34.077598 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:34.077599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:34.077599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:34.077665 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:34.077877 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:34.077878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:34.077878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:34.077976 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.078026 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.078249 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.078312 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:57:34.078539 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.069934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45265 (local address 127.1.74.67:42185)
0926 06:57:34.070088 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.070123 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.070627 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.070883 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.070887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.071144 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:34.071705 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.071711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.072408 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.072410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.073222 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.073224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.073622 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.073623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.073757 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.074209 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:34.074217 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:34.075594 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:34.077315 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:34.077320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:34.077322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:34.077528 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:34.077723 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:34.077723 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:34.077723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:34.077824 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:34.078028 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:34.078032 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:34.078143 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.078223 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.078341 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260926 06:57:34.078991 5862 heartbeater.cc:347] Connected to a master server at 127.1.74.126:45265
I20260926 06:57:34.079097 5862 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:34.079242 5862 heartbeater.cc:511] Master 127.1.74.126:45265 requested a full tablet report, sending...
I20260926 06:57:34.079638 5406 ts_manager.cc:194] Registered new tserver with Master: 2d75c5104f66442791bc9fc5ea16404a (127.1.74.67:45913)
I20260926 06:57:34.079986 5406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:42185
I20260926 06:57:34.087821 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:57:34.095007 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.089190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.1.74.126:45265)
0926 06:57:34.089261 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.089263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.089367 (+ 104us) server_negotiation.cc:407] Connection header received
0926 06:57:34.089445 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.089446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.089456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.089528 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.090120 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.090635 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.091460 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.091706 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.091772 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.092923 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.092924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.092924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:34.092928 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.094086 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:34.094378 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:34.094379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:34.094379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:34.094425 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:34.094567 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:34.094567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:34.094568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:34.094680 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.094776 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.094847 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.094905 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:57:34.096529 5406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:34.098076 5406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:34.104637 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.100516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.1.74.66:42927)
0926 06:57:34.100775 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.100780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.100789 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:34.100828 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.100830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.100845 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.100917 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:34.101079 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.102063 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.103722 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.104183 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.104288 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.104348 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.104515 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.104556 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":87,"threads_started":1}
I20260926 06:57:34.104642 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.100304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:45913 (local address 127.0.0.1:43514)
0926 06:57:34.100473 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.100529 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.100787 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.101060 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.101061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.101063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:34.101128 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.101129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.102500 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.102501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.103607 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.103609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.104442 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.104443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.104495 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.104497 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.104526 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.104559 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":34,"thread_start_us":54,"threads_started":1}
I20260926 06:57:34.104748 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.100322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:39279 (local address 127.0.0.1:55616)
0926 06:57:34.100679 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.100689 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.100741 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.101099 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.101100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.101101 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:34.101134 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.101135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.102627 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.102628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.103443 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.103445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.104331 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.104333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.104398 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.104404 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.104453 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.104505 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":50,"threads_started":1}
I20260926 06:57:34.104645 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.100304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:42927 (local address 127.0.0.1:34654)
0926 06:57:34.100565 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.100577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.100641 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.100922 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.100925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.100928 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:34.100995 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.100997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.102155 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.102155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.103640 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.103641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.104293 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.104294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.104383 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.104397 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.104497 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.104540 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260926 06:57:34.104837 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.100577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.1.74.65:39279)
0926 06:57:34.100913 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.100917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.100925 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:34.100961 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.100962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.100977 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.101345 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:34.101417 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.102532 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.103525 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.104230 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.104600 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.104665 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.104691 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.104731 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":79,"threads_started":1}
I20260926 06:57:34.105468 5663 tablet_service.cc:1467] Processing CreateTablet for tablet c318ef5d5122426db5b91267e8321e65 (DEFAULT_TABLE table=test-table [id=3485cf9a44124770aacdc9d0d09b5110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:34.105608 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.100691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.1.74.67:45913)
0926 06:57:34.100885 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.100890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.100900 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:34.100941 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.100943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.100958 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.101202 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:34.101277 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.102385 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.103685 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.104360 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.105408 (+ 1048us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.105467 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.105492 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.105533 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
I20260926 06:57:34.105506 5530 tablet_service.cc:1467] Processing CreateTablet for tablet c318ef5d5122426db5b91267e8321e65 (DEFAULT_TABLE table=test-table [id=3485cf9a44124770aacdc9d0d09b5110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:34.105824 5663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c318ef5d5122426db5b91267e8321e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:34.105842 5530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c318ef5d5122426db5b91267e8321e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:34.106155 5796 tablet_service.cc:1467] Processing CreateTablet for tablet c318ef5d5122426db5b91267e8321e65 (DEFAULT_TABLE table=test-table [id=3485cf9a44124770aacdc9d0d09b5110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:34.106442 5796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c318ef5d5122426db5b91267e8321e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:34.108460 5876 tablet_bootstrap.cc:492] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: Bootstrap starting.
I20260926 06:57:34.109124 5877 tablet_bootstrap.cc:492] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b: Bootstrap starting.
I20260926 06:57:34.109197 5876 tablet_bootstrap.cc:654] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:34.109570 5878 tablet_bootstrap.cc:492] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: Bootstrap starting.
I20260926 06:57:34.109777 5877 tablet_bootstrap.cc:654] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:34.109817 5876 log.cc:824] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:34.110213 5878 tablet_bootstrap.cc:654] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:34.110384 5877 log.cc:824] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:34.110702 5876 tablet_bootstrap.cc:492] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: No bootstrap required, opened a new log
I20260926 06:57:34.110790 5876 ts_tablet_manager.cc:1397] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:34.110790 5878 log.cc:824] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:34.111177 5877 tablet_bootstrap.cc:492] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b: No bootstrap required, opened a new log
I20260926 06:57:34.111248 5877 ts_tablet_manager.cc:1397] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:34.111892 5878 tablet_bootstrap.cc:492] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: No bootstrap required, opened a new log
I20260926 06:57:34.111963 5878 ts_tablet_manager.cc:1397] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:34.112689 5877 raft_consensus.cc:348] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } }
I20260926 06:57:34.112844 5877 raft_consensus.cc:374] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:34.112887 5877 raft_consensus.cc:729] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe44a70684b4056a154a51ff341333b, State: Initialized, Role: FOLLOWER
I20260926 06:57:34.112895 5876 raft_consensus.cc:348] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } }
I20260926 06:57:34.113055 5876 raft_consensus.cc:374] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:34.113089 5876 raft_consensus.cc:729] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb2df5f275045c6b055228366697308, State: Initialized, Role: FOLLOWER
I20260926 06:57:34.113015 5877 consensus_queue.cc:260] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } }
I20260926 06:57:34.113202 5876 consensus_queue.cc:260] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } }
I20260926 06:57:34.113322 5877 ts_tablet_manager.cc:1428] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:57:34.113443 5596 heartbeater.cc:503] Master 127.1.74.126:45265 was elected leader, sending a full tablet report...
I20260926 06:57:34.113782 5729 heartbeater.cc:503] Master 127.1.74.126:45265 was elected leader, sending a full tablet report...
I20260926 06:57:34.113763 5876 ts_tablet_manager.cc:1428] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:57:34.114019 5878 raft_consensus.cc:348] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } }
I20260926 06:57:34.114157 5878 raft_consensus.cc:374] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:34.114212 5878 raft_consensus.cc:729] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d75c5104f66442791bc9fc5ea16404a, State: Initialized, Role: FOLLOWER
I20260926 06:57:34.114351 5878 consensus_queue.cc:260] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } }
I20260926 06:57:34.114611 5878 ts_tablet_manager.cc:1428] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:57:34.114637 5862 heartbeater.cc:503] Master 127.1.74.126:45265 was elected leader, sending a full tablet report...
W20260926 06:57:34.175020 5730 tablet.cc:2369] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:34.253677 5882 raft_consensus.cc:482] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:34.253799 5882 raft_consensus.cc:504] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } }
I20260926 06:57:34.254217 5882 leader_election.cc:288] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fb2df5f275045c6b055228366697308 (127.1.74.66:42927), 2d75c5104f66442791bc9fc5ea16404a (127.1.74.67:45913)
I20260926 06:57:34.257038 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.254442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:50747 (local address 127.1.74.66:42927)
0926 06:57:34.254483 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.254485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.254495 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:34.254598 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.254599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.254609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.254672 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:34.254827 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.255356 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.256316 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.256685 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.256778 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.256788 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.256940 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.256979 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:57:34.257033 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.254390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:42927 (local address 127.1.74.65:50747)
0926 06:57:34.254445 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.254457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.254502 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.254676 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.254677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.254679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:34.254750 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.254751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.255433 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.255434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.256190 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.256192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.256769 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.256770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.256855 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.256858 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.256920 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.256959 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 06:57:34.257097 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.254498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:39857 (local address 127.1.74.67:45913)
0926 06:57:34.254642 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.254644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.254654 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:34.254712 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.254713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.254724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.254803 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:34.254955 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.255421 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.256370 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.256748 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.256836 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.256845 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.256988 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.257020 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260926 06:57:34.257141 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.254429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:45913 (local address 127.1.74.65:39857)
0926 06:57:34.254572 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.254582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.254641 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.254806 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.254807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.254808 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:34.254884 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.254885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.255512 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.255513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.256276 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:34.256278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.256835 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.256836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:34.256914 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.256916 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.256984 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.257021 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260926 06:57:34.257520 5683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c318ef5d5122426db5b91267e8321e65" candidate_uuid: "afe44a70684b4056a154a51ff341333b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb2df5f275045c6b055228366697308" is_pre_election: true
I20260926 06:57:34.257539 5816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c318ef5d5122426db5b91267e8321e65" candidate_uuid: "afe44a70684b4056a154a51ff341333b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d75c5104f66442791bc9fc5ea16404a" is_pre_election: true
I20260926 06:57:34.257680 5683 raft_consensus.cc:2454] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe44a70684b4056a154a51ff341333b in term 0.
I20260926 06:57:34.257692 5816 raft_consensus.cc:2454] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe44a70684b4056a154a51ff341333b in term 0.
I20260926 06:57:34.257922 5486 leader_election.cc:302] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d75c5104f66442791bc9fc5ea16404a, afe44a70684b4056a154a51ff341333b; no voters:
I20260926 06:57:34.258112 5882 raft_consensus.cc:2781] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:34.258150 5882 raft_consensus.cc:482] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:34.258164 5882 raft_consensus.cc:3037] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:34.258802 5882 raft_consensus.cc:504] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } }
I20260926 06:57:34.258941 5882 leader_election.cc:288] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [CANDIDATE]: Term 1 election: Requested vote from peers 8fb2df5f275045c6b055228366697308 (127.1.74.66:42927), 2d75c5104f66442791bc9fc5ea16404a (127.1.74.67:45913)
I20260926 06:57:34.259091 5683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c318ef5d5122426db5b91267e8321e65" candidate_uuid: "afe44a70684b4056a154a51ff341333b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb2df5f275045c6b055228366697308"
I20260926 06:57:34.259104 5816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c318ef5d5122426db5b91267e8321e65" candidate_uuid: "afe44a70684b4056a154a51ff341333b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d75c5104f66442791bc9fc5ea16404a"
I20260926 06:57:34.259172 5816 raft_consensus.cc:3037] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:34.259186 5683 raft_consensus.cc:3037] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:34.259867 5816 raft_consensus.cc:2454] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe44a70684b4056a154a51ff341333b in term 1.
I20260926 06:57:34.260000 5683 raft_consensus.cc:2454] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe44a70684b4056a154a51ff341333b in term 1.
I20260926 06:57:34.260039 5486 leader_election.cc:302] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d75c5104f66442791bc9fc5ea16404a, afe44a70684b4056a154a51ff341333b; no voters:
I20260926 06:57:34.260135 5882 raft_consensus.cc:2781] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:34.260315 5882 raft_consensus.cc:686] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [term 1 LEADER]: Becoming Leader. State: Replica: afe44a70684b4056a154a51ff341333b, State: Running, Role: LEADER
I20260926 06:57:34.260413 5882 consensus_queue.cc:237] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } }
I20260926 06:57:34.261243 5404 catalog_manager.cc:5495] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b reported cstate change: term changed from 0 to 1, leader changed from <none> to afe44a70684b4056a154a51ff341333b (127.1.74.65). New cstate: current_term: 1 leader_uuid: "afe44a70684b4056a154a51ff341333b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afe44a70684b4056a154a51ff341333b" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 39279 } health_report { overall_health: HEALTHY } } }
W20260926 06:57:34.269318 5597 tablet.cc:2369] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:34.289584 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.286726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.1.74.65:39279)
0926 06:57:34.286798 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.286800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.286819 (+ 19us) server_negotiation.cc:407] Connection header received
0926 06:57:34.286861 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.286862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.286885 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.286973 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:34.287132 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.287775 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.288625 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:34.288838 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:34.288914 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:34.289098 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:34.289209 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:34.289361 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.289445 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.289464 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:57:34.291448 5683 raft_consensus.cc:1261] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [term 1 FOLLOWER]: Refusing update from remote peer afe44a70684b4056a154a51ff341333b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:34.291865 5816 raft_consensus.cc:1261] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [term 1 FOLLOWER]: Refusing update from remote peer afe44a70684b4056a154a51ff341333b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:34.291872 5882 consensus_queue.cc:1035] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fb2df5f275045c6b055228366697308" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:34.292148 5882 consensus_queue.cc:1035] T c318ef5d5122426db5b91267e8321e65 P afe44a70684b4056a154a51ff341333b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d75c5104f66442791bc9fc5ea16404a" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:34.294445 5891 mvcc.cc:205] Tried to move back new op lower bound from 7333502379174862848 to 7333502379051245568. Current Snapshot: MvccSnapshot[applied={T|T < 7333502379174862848}]
I20260926 06:57:34.294518 5890 mvcc.cc:205] Tried to move back new op lower bound from 7333502379174862848 to 7333502379051245568. Current Snapshot: MvccSnapshot[applied={T|T < 7333502379174862848}]
I20260926 06:57:34.296890 5405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:34.296979 5405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:34.297730 5405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f6efa1b5b724dd2aa212986168cd19d: Sending DeleteTablet for 3 replicas of tablet c318ef5d5122426db5b91267e8321e65
I20260926 06:57:34.298040 5530 tablet_service.cc:1514] Processing DeleteTablet for tablet c318ef5d5122426db5b91267e8321e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:34 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:55616
I20260926 06:57:34.298045 5796 tablet_service.cc:1514] Processing DeleteTablet for tablet c318ef5d5122426db5b91267e8321e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:34 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:43514
I20260926 06:57:34.298066 5663 tablet_service.cc:1514] Processing DeleteTablet for tablet c318ef5d5122426db5b91267e8321e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:34 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:34654
I20260926 06:57:34.298358 5901 tablet_replica.cc:330] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: stopping tablet replica
I20260926 06:57:34.298517 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 5468
I20260926 06:57:34.299813 5901 raft_consensus.cc:2229] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:34.299933 5901 raft_consensus.cc:2258] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:34.300096 5900 tablet_replica.cc:330] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: stopping tablet replica
I20260926 06:57:34.300184 5900 raft_consensus.cc:2229] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:34.300274 5900 raft_consensus.cc:2258] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:34.300501 5901 ts_tablet_manager.cc:1905] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:34.300729 5900 ts_tablet_manager.cc:1905] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:34.302179 5900 ts_tablet_manager.cc:1918] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:34.302248 5900 log.cc:1196] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/c318ef5d5122426db5b91267e8321e65
I20260926 06:57:34.302214 5901 ts_tablet_manager.cc:1918] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:34.302299 5901 log.cc:1196] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/c318ef5d5122426db5b91267e8321e65
I20260926 06:57:34.302508 5900 ts_tablet_manager.cc:1939] T c318ef5d5122426db5b91267e8321e65 P 8fb2df5f275045c6b055228366697308: Deleting consensus metadata
I20260926 06:57:34.302572 5901 ts_tablet_manager.cc:1939] T c318ef5d5122426db5b91267e8321e65 P 2d75c5104f66442791bc9fc5ea16404a: Deleting consensus metadata
I20260926 06:57:34.302912 5390 catalog_manager.cc:4841] TS 8fb2df5f275045c6b055228366697308 (127.1.74.66:42927): tablet c318ef5d5122426db5b91267e8321e65 (table test-table [id=3485cf9a44124770aacdc9d0d09b5110]) successfully deleted
I20260926 06:57:34.303083 5393 catalog_manager.cc:4841] TS 2d75c5104f66442791bc9fc5ea16404a (127.1.74.67:45913): tablet c318ef5d5122426db5b91267e8321e65 (table test-table [id=3485cf9a44124770aacdc9d0d09b5110]) successfully deleted
W20260926 06:57:34.304507 5391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:39279 (error 108)
I20260926 06:57:34.304657 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 5601
I20260926 06:57:34.305413 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.305263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:39279 (local address 127.0.0.1:55628)
0926 06:57:34.305331 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.305373 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 06:57:34.305550 5391 catalog_manager.cc:4584] TS afe44a70684b4056a154a51ff341333b (127.1.74.65:39279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c318ef5d5122426db5b91267e8321e65: Network error: Client connection negotiation failed: client connection to 127.1.74.65:39279: connect: Connection refused (error 111)
I20260926 06:57:34.310107 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 5734
I20260926 06:57:34.314785 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 5375
2026-09-26T06:57:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2011 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T06:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:34Z Disabled control of system clock
I20260926 06:57:34.341826 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:41029
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:41029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 06:57:34.424124 5906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:34.424324 5906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:34.424343 5906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:34.425407 5906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:34.425442 5906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:34.425455 5906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:34.425467 5906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:34.425479 5906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:34.426715 5906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:41029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:34.426908 5906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:34.427126 5906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:34.431778 5914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:34.431788 5912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:34.431804 5911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:34.432149 5906 server_base.cc:1034] running on GCE node
I20260926 06:57:34.432407 5906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:34.432683 5906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:34.433841 5906 hybrid_clock.cc:648] HybridClock initialized: now 1790405854433824 us; error 43 us; skew 500 ppm
I20260926 06:57:34.435077 5906 webserver.cc:460] Webserver started at http://127.1.74.126:32799/ using document root <none> and password file <none>
I20260926 06:57:34.435424 5906 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:34.435493 5906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:34.435629 5906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:34.436651 5906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "181db9411dcf4bdebebdc276ed2892b7"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "4011e5a95866a1c9a57057f5aeae51a2"
server_key_iv: "a008b16e75645ffc6edd2f0a74ab9376"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.437012 5906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "181db9411dcf4bdebebdc276ed2892b7"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "4011e5a95866a1c9a57057f5aeae51a2"
server_key_iv: "a008b16e75645ffc6edd2f0a74ab9376"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.438463 5906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:34.439172 5920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.439348 5906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:34.439457 5906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "181db9411dcf4bdebebdc276ed2892b7"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "4011e5a95866a1c9a57057f5aeae51a2"
server_key_iv: "a008b16e75645ffc6edd2f0a74ab9376"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.439543 5906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:34.442819 5906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:34.443090 5906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:34.443235 5906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:34.447037 5906 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:41029
I20260926 06:57:34.447093 5972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:41029 every 8 connection(s)
I20260926 06:57:34.447474 5906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:34.448102 5973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:34.450730 5973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7: Bootstrap starting.
I20260926 06:57:34.451310 5973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:34.451591 5973 log.cc:824] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:34.452206 5973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7: No bootstrap required, opened a new log
I20260926 06:57:34.453677 5973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181db9411dcf4bdebebdc276ed2892b7" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41029 } }
I20260926 06:57:34.453809 5973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:34.453830 5973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181db9411dcf4bdebebdc276ed2892b7, State: Initialized, Role: FOLLOWER
I20260926 06:57:34.453963 5973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181db9411dcf4bdebebdc276ed2892b7" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41029 } }
I20260926 06:57:34.454039 5973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:34.454061 5973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:34.454087 5973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:34.454564 5973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181db9411dcf4bdebebdc276ed2892b7" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41029 } }
I20260926 06:57:34.454643 5973 leader_election.cc:302] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181db9411dcf4bdebebdc276ed2892b7; no voters:
I20260926 06:57:34.454764 5973 leader_election.cc:288] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:34.454833 5976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:34.455008 5976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [term 1 LEADER]: Becoming Leader. State: Replica: 181db9411dcf4bdebebdc276ed2892b7, State: Running, Role: LEADER
I20260926 06:57:34.455021 5973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:34.455092 5976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181db9411dcf4bdebebdc276ed2892b7" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41029 } }
I20260926 06:57:34.455677 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 5906
I20260926 06:57:34.455792 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:34.456003 1321 external_mini_cluster.cc:1428] Setting key 6a3bcf83724c8be38f5a7ddf84847b88
I20260926 06:57:34.456449 5978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181db9411dcf4bdebebdc276ed2892b7. Latest consensus state: current_term: 1 leader_uuid: "181db9411dcf4bdebebdc276ed2892b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181db9411dcf4bdebebdc276ed2892b7" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41029 } } }
I20260926 06:57:34.456542 5978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:34.456631 5977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181db9411dcf4bdebebdc276ed2892b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181db9411dcf4bdebebdc276ed2892b7" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 41029 } } }
I20260926 06:57:34.456863 5977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7 [sys.catalog]: This master's current role is: LEADER
W20260926 06:57:34.457127 5993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:34.457186 5993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:34.457759 5994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:34.457964 5994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:34.458602 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.456464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.1.74.126:41029)
0926 06:57:34.456947 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.456956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.457025 (+ 69us) server_negotiation.cc:407] Connection header received
0926 06:57:34.457225 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.457232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.457292 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.457420 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.458128 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.458131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.458132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.458137 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.458195 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.458205 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.458278 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.458346 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.458404 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":68,"threads_started":1}
I20260926 06:57:34.459393 5994 catalog_manager.cc:1347] Generated new cluster ID: 8929bbb3f06c4c79990d80de6864d9ce
I20260926 06:57:34.459446 5994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:34.469885 5994 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:34.470050 5994 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:34.473948 5994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7: Generated new TSK 0
I20260926 06:57:34.474121 5994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:57:34.479300 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:41029
--builtin_ntp_servers=127.1.74.84:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 06:57:34.564929 5997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:34.565161 5997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:34.565191 5997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:34.566342 5997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:34.566381 5997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:34.566407 5997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:34.567768 5997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:41029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:34.567991 5997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:34.568311 5997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:34.574160 6003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:34.574138 6005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:34.574143 6002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:34.574426 5997 server_base.cc:1034] running on GCE node
I20260926 06:57:34.574710 5997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:34.574993 5997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:34.576148 5997 hybrid_clock.cc:648] HybridClock initialized: now 1790405854576136 us; error 27 us; skew 500 ppm
I20260926 06:57:34.577510 5997 webserver.cc:460] Webserver started at http://127.1.74.65:45519/ using document root <none> and password file <none>
I20260926 06:57:34.577788 5997 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:34.577834 5997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:34.577972 5997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:34.579046 5997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "3a0fad4bd9174c9abd4d579866fe73db"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "5311b4af7f8692d85ab01bfc4d03d6e6"
server_key_iv: "3fec271f862ae986c57e6e8bfe1092f7"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.579457 5997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "3a0fad4bd9174c9abd4d579866fe73db"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "5311b4af7f8692d85ab01bfc4d03d6e6"
server_key_iv: "3fec271f862ae986c57e6e8bfe1092f7"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.580832 5997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:34.581579 6011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.581729 5997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:34.581831 5997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "3a0fad4bd9174c9abd4d579866fe73db"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "5311b4af7f8692d85ab01bfc4d03d6e6"
server_key_iv: "3fec271f862ae986c57e6e8bfe1092f7"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.581903 5997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:34.585388 5997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:34.585659 5997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:34.585800 5997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:34.586025 5997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:34.586404 5997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:34.586439 5997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.586459 5997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:34.586472 5997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.592718 5997 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:40537
I20260926 06:57:34.592782 6124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:40537 every 8 connection(s)
I20260926 06:57:34.593078 5997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:34.594267 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 5997
I20260926 06:57:34.594344 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:34.594511 1321 external_mini_cluster.cc:1428] Setting key 793b9e8555acb8f2709a31d66729fccc
I20260926 06:57:34.595623 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:41029
--builtin_ntp_servers=127.1.74.84:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 06:57:34.596397 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.593811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:43269 (local address 127.1.74.126:41029)
0926 06:57:34.593885 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.593887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.594076 (+ 189us) server_negotiation.cc:407] Connection header received
0926 06:57:34.594660 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.594661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.594668 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.594752 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.596116 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.596118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.596118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.596123 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.596141 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.596146 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.596216 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.596282 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.596298 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:57:34.596510 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.593767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41029 (local address 127.1.74.65:43269)
0926 06:57:34.593963 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.593995 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.594514 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.594807 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.594810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.594818 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:34.595966 (+ 1148us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:57:34.595966 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:34.595995 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:57:34.596022 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:34.596214 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:34.596216 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:34.596218 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.596269 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.596321 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260926 06:57:34.596976 6125 heartbeater.cc:347] Connected to a master server at 127.1.74.126:41029
I20260926 06:57:34.597084 6125 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:34.597234 6125 heartbeater.cc:511] Master 127.1.74.126:41029 requested a full tablet report, sending...
I20260926 06:57:34.597831 5937 ts_manager.cc:194] Registered new tserver with Master: 3a0fad4bd9174c9abd4d579866fe73db (127.1.74.65:40537)
W20260926 06:57:34.676165 6128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:34.676367 6128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:34.676388 6128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:34.677449 6128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:34.677482 6128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:34.677507 6128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:34.678753 6128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:41029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:34.678941 6128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:34.679171 6128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:34.684161 6136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:34.684173 6133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:34.684293 6128 server_base.cc:1034] running on GCE node
W20260926 06:57:34.684161 6134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:34.684777 6128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:34.685037 6128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:34.686209 6128 hybrid_clock.cc:648] HybridClock initialized: now 1790405854686186 us; error 57 us; skew 500 ppm
I20260926 06:57:34.687520 6128 webserver.cc:460] Webserver started at http://127.1.74.66:45593/ using document root <none> and password file <none>
I20260926 06:57:34.687775 6128 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:34.687850 6128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:34.688000 6128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:34.689088 6128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "4954b219d7634d68b40e31b5657a0b2a"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "184a44264b44ff19dc4f6b4e62fe8578"
server_key_iv: "65b73130feaf784e1cc2d91ea1151503"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.689486 6128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "4954b219d7634d68b40e31b5657a0b2a"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "184a44264b44ff19dc4f6b4e62fe8578"
server_key_iv: "65b73130feaf784e1cc2d91ea1151503"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.690857 6128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:34.691615 6142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.691789 6128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.691891 6128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "4954b219d7634d68b40e31b5657a0b2a"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "184a44264b44ff19dc4f6b4e62fe8578"
server_key_iv: "65b73130feaf784e1cc2d91ea1151503"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.691963 6128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:34.695453 6128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:34.695739 6128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:34.695887 6128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:34.696099 6128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:34.696414 6128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:34.696467 6128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.696504 6128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:34.696533 6128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.702250 6128 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:42931
I20260926 06:57:34.702287 6255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:42931 every 8 connection(s)
I20260926 06:57:34.702651 6128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:34.705762 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.703440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:37117 (local address 127.1.74.126:41029)
0926 06:57:34.703485 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.703487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.703757 (+ 270us) server_negotiation.cc:407] Connection header received
0926 06:57:34.704291 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.704292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.704299 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.704359 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.705505 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.705507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.705507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.705512 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.705528 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.705533 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.705588 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.705677 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.705693 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:57:34.705996 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.703461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41029 (local address 127.1.74.66:37117)
0926 06:57:34.703636 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.703670 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.704165 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.704438 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.704442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.704450 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:34.705350 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:57:34.705350 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:34.705380 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:57:34.705408 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:34.705583 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:34.705587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:34.705591 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.705688 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.705751 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260926 06:57:34.706449 6256 heartbeater.cc:347] Connected to a master server at 127.1.74.126:41029
I20260926 06:57:34.706523 6256 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:34.706693 6256 heartbeater.cc:511] Master 127.1.74.126:41029 requested a full tablet report, sending...
I20260926 06:57:34.707027 5937 ts_manager.cc:194] Registered new tserver with Master: 4954b219d7634d68b40e31b5657a0b2a (127.1.74.66:42931)
I20260926 06:57:34.710623 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 6128
I20260926 06:57:34.710705 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:34.710842 1321 external_mini_cluster.cc:1428] Setting key 32606e0c616ed533f665416448d4af52
I20260926 06:57:34.711813 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:41029
--builtin_ntp_servers=127.1.74.84:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 06:57:34.791597 6259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:34.791834 6259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:34.791872 6259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:34.793052 6259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:34.793115 6259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:34.793157 6259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:34.794571 6259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:41029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:34.794827 6259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:34.795130 6259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:34.800149 6264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:34.800148 6267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:34.800156 6265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:34.800560 6259 server_base.cc:1034] running on GCE node
I20260926 06:57:34.800734 6259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:34.800927 6259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:34.802064 6259 hybrid_clock.cc:648] HybridClock initialized: now 1790405854802046 us; error 31 us; skew 500 ppm
I20260926 06:57:34.803344 6259 webserver.cc:460] Webserver started at http://127.1.74.67:40301/ using document root <none> and password file <none>
I20260926 06:57:34.803582 6259 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:34.803630 6259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:34.803762 6259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:34.804840 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "aa15cb94e2e347e5a4c6ece756c0c98e"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "80cd0df0d337e6c3cbb2510920ba8377"
server_key_iv: "cdfc2e225a418bbc1cdd98c7846dfcaf"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.805212 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "aa15cb94e2e347e5a4c6ece756c0c98e"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "80cd0df0d337e6c3cbb2510920ba8377"
server_key_iv: "cdfc2e225a418bbc1cdd98c7846dfcaf"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.806668 6259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:34.807446 6273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.807621 6259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:34.807689 6259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "aa15cb94e2e347e5a4c6ece756c0c98e"
format_stamp: "Formatted at 2026-09-26 06:57:34 on dist-test-slave-sfvt"
server_key: "80cd0df0d337e6c3cbb2510920ba8377"
server_key_iv: "cdfc2e225a418bbc1cdd98c7846dfcaf"
server_key_version: "encryptionkey@0"
I20260926 06:57:34.807780 6259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:34.811065 6259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:34.811388 6259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:34.811545 6259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:34.811755 6259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:34.812065 6259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:34.812100 6259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.812119 6259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:34.812132 6259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:34.817834 6259 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:45945
I20260926 06:57:34.817864 6386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:45945 every 8 connection(s)
I20260926 06:57:34.818197 6259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:57:34.821208 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.818947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:54453 (local address 127.1.74.126:41029)
0926 06:57:34.819006 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.819008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.819297 (+ 289us) server_negotiation.cc:407] Connection header received
0926 06:57:34.819758 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.819759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.819763 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.819824 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.821022 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.821022 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.821023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.821026 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.821036 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.821040 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.821081 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.821144 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.821156 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:57:34.821409 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.818918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:41029 (local address 127.1.74.67:54453)
0926 06:57:34.819112 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.819157 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.819662 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.819888 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.819891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.819900 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:34.820885 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:57:34.820886 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:34.820915 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:57:34.820943 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:34.821081 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:34.821083 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:34.821086 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.821136 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.821184 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260926 06:57:34.821939 6387 heartbeater.cc:347] Connected to a master server at 127.1.74.126:41029
I20260926 06:57:34.822038 6387 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:34.822196 6387 heartbeater.cc:511] Master 127.1.74.126:41029 requested a full tablet report, sending...
I20260926 06:57:34.822546 5937 ts_manager.cc:194] Registered new tserver with Master: aa15cb94e2e347e5a4c6ece756c0c98e (127.1.74.67:45945)
I20260926 06:57:34.826200 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 6259
I20260926 06:57:34.826277 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:34.826483 1321 external_mini_cluster.cc:1428] Setting key aae727daf91dcce9e1987b230a90a95d
I20260926 06:57:34.827369 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:57:34.829721 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.828754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.1.74.126:41029)
0926 06:57:34.828792 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.828794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.828893 (+ 99us) server_negotiation.cc:407] Connection header received
0926 06:57:34.828940 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.828941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.828945 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.828988 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.829579 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.829579 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.829579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.829582 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.829593 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.829596 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.829634 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.829665 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.829676 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:57:34.831406 5937 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:34.832988 5937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:34.837124 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.835212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40537 (local address 127.0.0.1:38532)
0926 06:57:34.835752 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.835763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.835822 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.836048 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.836053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.836060 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:34.836732 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:57:34.836732 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:34.836749 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:57:34.836772 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:34.836965 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:34.836971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:34.836974 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.837043 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.837056 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260926 06:57:34.837143 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.835335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.1.74.65:40537)
0926 06:57:34.835514 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.835519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.835899 (+ 380us) server_negotiation.cc:407] Connection header received
0926 06:57:34.835929 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.835932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.835948 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.836032 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.836836 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.836844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.836844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.836849 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.836882 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.836905 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.836964 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.837051 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.837065 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260926 06:57:34.837251 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.835325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.1.74.66:42931)
0926 06:57:34.835499 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.835542 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.835692 (+ 150us) server_negotiation.cc:407] Connection header received
0926 06:57:34.835746 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.835748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.835761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.835842 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.836920 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.836927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.836927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.836932 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.836966 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.836987 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.837059 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.837155 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.837171 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260926 06:57:34.837239 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.835212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:42931 (local address 127.0.0.1:47806)
0926 06:57:34.835573 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.835614 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.835675 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.836063 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.836064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.836067 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:34.836802 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:57:34.836802 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:34.836810 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:57:34.836825 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:34.837068 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:34.837069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:34.837070 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.837145 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.837160 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":115,"threads_started":1}
I20260926 06:57:34.837757 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.835454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:45945 (local address 127.0.0.1:53210)
0926 06:57:34.835906 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.835915 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.835949 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.836521 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.836521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.836524 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:34.837353 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:57:34.837353 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:34.837359 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:57:34.837370 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:34.837608 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:34.837609 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:34.837610 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.837673 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.837686 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":68,"threads_started":1}
I20260926 06:57:34.837797 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.835725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.1.74.67:45945)
0926 06:57:34.836387 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.836390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.836398 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:34.836418 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.836419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.836424 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.836520 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.837439 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.837444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.837444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.837449 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.837486 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.837510 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.837578 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.837687 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.837701 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":104,"threads_started":1}
I20260926 06:57:34.838194 6190 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9bd24b9bf8467c940929d161c36791 (DEFAULT_TABLE table=test-table [id=5afea267c13e4c33abfcf9bc7305a85c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:34.838186 6059 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9bd24b9bf8467c940929d161c36791 (DEFAULT_TABLE table=test-table [id=5afea267c13e4c33abfcf9bc7305a85c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:34.838523 6190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9bd24b9bf8467c940929d161c36791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:34.838531 6059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9bd24b9bf8467c940929d161c36791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:34.841188 6401 tablet_bootstrap.cc:492] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: Bootstrap starting.
I20260926 06:57:34.841243 6402 tablet_bootstrap.cc:492] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db: Bootstrap starting.
I20260926 06:57:34.841750 6321 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9bd24b9bf8467c940929d161c36791 (DEFAULT_TABLE table=test-table [id=5afea267c13e4c33abfcf9bc7305a85c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:34.841944 6402 tablet_bootstrap.cc:654] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:34.842001 6401 tablet_bootstrap.cc:654] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:34.842075 6321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9bd24b9bf8467c940929d161c36791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:34.842242 6401 log.cc:824] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:34.842268 6402 log.cc:824] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:34.842958 6402 tablet_bootstrap.cc:492] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db: No bootstrap required, opened a new log
I20260926 06:57:34.842985 6401 tablet_bootstrap.cc:492] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: No bootstrap required, opened a new log
I20260926 06:57:34.843029 6402 ts_tablet_manager.cc:1397] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:34.843037 6401 ts_tablet_manager.cc:1397] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:34.845079 6402 raft_consensus.cc:348] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } }
I20260926 06:57:34.845232 6402 raft_consensus.cc:374] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:34.845255 6402 raft_consensus.cc:729] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0fad4bd9174c9abd4d579866fe73db, State: Initialized, Role: FOLLOWER
I20260926 06:57:34.845420 6405 tablet_bootstrap.cc:492] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e: Bootstrap starting.
I20260926 06:57:34.845398 6402 consensus_queue.cc:260] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } }
I20260926 06:57:34.845659 6402 ts_tablet_manager.cc:1428] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:57:34.845593 6401 raft_consensus.cc:348] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } }
I20260926 06:57:34.845729 6401 raft_consensus.cc:374] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:34.845757 6401 raft_consensus.cc:729] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4954b219d7634d68b40e31b5657a0b2a, State: Initialized, Role: FOLLOWER
I20260926 06:57:34.845857 6401 consensus_queue.cc:260] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } }
I20260926 06:57:34.845932 6125 heartbeater.cc:503] Master 127.1.74.126:41029 was elected leader, sending a full tablet report...
I20260926 06:57:34.846096 6401 ts_tablet_manager.cc:1428] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:57:34.846119 6256 heartbeater.cc:503] Master 127.1.74.126:41029 was elected leader, sending a full tablet report...
I20260926 06:57:34.846328 6405 tablet_bootstrap.cc:654] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:34.846771 6405 log.cc:824] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:34.847539 6405 tablet_bootstrap.cc:492] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e: No bootstrap required, opened a new log
I20260926 06:57:34.847602 6405 ts_tablet_manager.cc:1397] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:34.849474 6405 raft_consensus.cc:348] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } }
I20260926 06:57:34.849612 6405 raft_consensus.cc:374] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:34.849637 6405 raft_consensus.cc:729] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa15cb94e2e347e5a4c6ece756c0c98e, State: Initialized, Role: FOLLOWER
I20260926 06:57:34.849840 6405 consensus_queue.cc:260] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } }
I20260926 06:57:34.849936 6409 raft_consensus.cc:482] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:34.850013 6409 raft_consensus.cc:504] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } }
I20260926 06:57:34.850088 6405 ts_tablet_manager.cc:1428] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:57:34.850205 6387 heartbeater.cc:503] Master 127.1.74.126:41029 was elected leader, sending a full tablet report...
I20260926 06:57:34.850350 6409 leader_election.cc:288] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4954b219d7634d68b40e31b5657a0b2a (127.1.74.66:42931), 3a0fad4bd9174c9abd4d579866fe73db (127.1.74.65:40537)
I20260926 06:57:34.851743 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.850505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:42931 (local address 127.1.74.67:40545)
0926 06:57:34.850560 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.850572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.850612 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.850814 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.850815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.850819 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:34.851371 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:57:34.851372 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:34.851380 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:57:34.851393 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:34.851603 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:34.851604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:34.851605 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.851665 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.851680 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 06:57:34.851789 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.850607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:40545 (local address 127.1.74.66:42931)
0926 06:57:34.850651 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.850653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.850662 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:34.850685 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.850686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.850692 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.850753 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.851502 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.851504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.851504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.851508 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.851525 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.851529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.851583 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.851691 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.851704 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:57:34.852037 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.850767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:51829 (local address 127.1.74.65:40537)
0926 06:57:34.850805 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.850807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.850990 (+ 183us) server_negotiation.cc:407] Connection header received
0926 06:57:34.851091 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.851092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.851095 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.851124 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.851700 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.851701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.851702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.851705 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.851718 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.851721 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.851774 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.851987 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.851998 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:57:34.852123 6210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9bd24b9bf8467c940929d161c36791" candidate_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4954b219d7634d68b40e31b5657a0b2a" is_pre_election: true
I20260926 06:57:34.852260 6210 raft_consensus.cc:2454] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa15cb94e2e347e5a4c6ece756c0c98e in term 0.
I20260926 06:57:34.852463 6274 leader_election.cc:302] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4954b219d7634d68b40e31b5657a0b2a, aa15cb94e2e347e5a4c6ece756c0c98e; no voters:
I20260926 06:57:34.852479 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.850735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40537 (local address 127.1.74.67:51829)
0926 06:57:34.850931 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.850940 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:34.851020 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:34.851190 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:34.851190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:34.851193 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:34.851607 (+ 414us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:57:34.851608 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:34.851614 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:57:34.851625 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:34.851774 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:34.851775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:34.851775 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 06:57:34.852374 (+ 599us) client_negotiation.cc:240] Negotiation successful
0926 06:57:34.852390 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260926 06:57:34.852613 6410 raft_consensus.cc:2781] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:34.852661 6410 raft_consensus.cc:482] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:34.852695 6410 raft_consensus.cc:3037] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:34.852792 6079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9bd24b9bf8467c940929d161c36791" candidate_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0fad4bd9174c9abd4d579866fe73db" is_pre_election: true
I20260926 06:57:34.852897 6079 raft_consensus.cc:2454] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa15cb94e2e347e5a4c6ece756c0c98e in term 0.
I20260926 06:57:34.853428 6410 raft_consensus.cc:504] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } }
I20260926 06:57:34.853567 6410 leader_election.cc:288] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [CANDIDATE]: Term 1 election: Requested vote from peers 4954b219d7634d68b40e31b5657a0b2a (127.1.74.66:42931), 3a0fad4bd9174c9abd4d579866fe73db (127.1.74.65:40537)
I20260926 06:57:34.853719 6079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9bd24b9bf8467c940929d161c36791" candidate_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a0fad4bd9174c9abd4d579866fe73db"
I20260926 06:57:34.853722 6210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9bd24b9bf8467c940929d161c36791" candidate_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4954b219d7634d68b40e31b5657a0b2a"
I20260926 06:57:34.853811 6079 raft_consensus.cc:3037] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:34.853801 6210 raft_consensus.cc:3037] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:34.854650 6210 raft_consensus.cc:2454] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa15cb94e2e347e5a4c6ece756c0c98e in term 1.
I20260926 06:57:34.854627 6079 raft_consensus.cc:2454] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa15cb94e2e347e5a4c6ece756c0c98e in term 1.
I20260926 06:57:34.854802 6274 leader_election.cc:302] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4954b219d7634d68b40e31b5657a0b2a, aa15cb94e2e347e5a4c6ece756c0c98e; no voters:
I20260926 06:57:34.854883 6410 raft_consensus.cc:2781] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:34.854969 6410 raft_consensus.cc:686] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 1 LEADER]: Becoming Leader. State: Replica: aa15cb94e2e347e5a4c6ece756c0c98e, State: Running, Role: LEADER
I20260926 06:57:34.855041 6410 consensus_queue.cc:237] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } }
I20260926 06:57:34.855885 5936 catalog_manager.cc:5495] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e reported cstate change: term changed from 0 to 1, leader changed from <none> to aa15cb94e2e347e5a4c6ece756c0c98e (127.1.74.67). New cstate: current_term: 1 leader_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa15cb94e2e347e5a4c6ece756c0c98e" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 } health_report { overall_health: UNKNOWN } } }
I20260926 06:57:34.862454 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.861326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.1.74.67:45945)
0926 06:57:34.861384 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:34.861386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:34.861398 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:57:34.861476 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:34.861476 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:34.861486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:34.861532 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:34.862268 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:34.862269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:34.862269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:34.862272 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:34.862285 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:34.862288 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:34.862339 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:34.862396 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 06:57:34.862407 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:57:34.864203 6210 raft_consensus.cc:1261] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [term 1 FOLLOWER]: Refusing update from remote peer aa15cb94e2e347e5a4c6ece756c0c98e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:34.864202 6079 raft_consensus.cc:1261] T 5f9bd24b9bf8467c940929d161c36791 P 3a0fad4bd9174c9abd4d579866fe73db [term 1 FOLLOWER]: Refusing update from remote peer aa15cb94e2e347e5a4c6ece756c0c98e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:34.864584 6410 consensus_queue.cc:1035] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a0fad4bd9174c9abd4d579866fe73db" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:34.864691 6409 consensus_queue.cc:1035] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4954b219d7634d68b40e31b5657a0b2a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:34.866956 6414 mvcc.cc:205] Tried to move back new op lower bound from 7333502381521031168 to 7333502381486886912. Current Snapshot: MvccSnapshot[applied={T|T < 7333502381521031168}]
I20260926 06:57:34.868888 6415 mvcc.cc:205] Tried to move back new op lower bound from 7333502381521031168 to 7333502381486886912. Current Snapshot: MvccSnapshot[applied={T|T < 7333502381521031168}]
I20260926 06:57:34.869243 5937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:34.869335 5937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:34.870018 5937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 181db9411dcf4bdebebdc276ed2892b7: Sending DeleteTablet for 3 replicas of tablet 5f9bd24b9bf8467c940929d161c36791
I20260926 06:57:34.870361 6059 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9bd24b9bf8467c940929d161c36791 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:34 UTC) from {username='slave'} at 127.0.0.1:38532
I20260926 06:57:34.870370 6321 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9bd24b9bf8467c940929d161c36791 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:34 UTC) from {username='slave'} at 127.0.0.1:53210
I20260926 06:57:34.870580 6190 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9bd24b9bf8467c940929d161c36791 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:34 UTC) from {username='slave'} at 127.0.0.1:47806
I20260926 06:57:34.870652 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 5997
I20260926 06:57:34.872275 6425 tablet_replica.cc:330] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: stopping tablet replica
I20260926 06:57:34.872360 6425 raft_consensus.cc:2229] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:34.872434 6425 raft_consensus.cc:2258] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:34.872877 6425 ts_tablet_manager.cc:1905] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:34.874820 6425 ts_tablet_manager.cc:1918] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:34.874958 6425 log.cc:1196] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/5f9bd24b9bf8467c940929d161c36791
I20260926 06:57:34.875355 6425 ts_tablet_manager.cc:1939] T 5f9bd24b9bf8467c940929d161c36791 P 4954b219d7634d68b40e31b5657a0b2a: Deleting consensus metadata
W20260926 06:57:34.875407 5922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:40537 (error 108)
I20260926 06:57:34.875859 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 6128
I20260926 06:57:34.875886 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.875743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:40537 (local address 127.0.0.1:38536)
0926 06:57:34.875799 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.875849 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 06:57:34.875890 5921 catalog_manager.cc:4841] TS 4954b219d7634d68b40e31b5657a0b2a (127.1.74.66:42931): tablet 5f9bd24b9bf8467c940929d161c36791 (table test-table [id=5afea267c13e4c33abfcf9bc7305a85c]) successfully deleted
W20260926 06:57:34.878006 5922 catalog_manager.cc:4584] TS 3a0fad4bd9174c9abd4d579866fe73db (127.1.74.65:40537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f9bd24b9bf8467c940929d161c36791: Network error: Client connection negotiation failed: client connection to 127.1.74.65:40537: connect: Connection refused (error 111)
I20260926 06:57:34.881019 6424 tablet_replica.cc:330] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e: stopping tablet replica
I20260926 06:57:34.881160 6424 raft_consensus.cc:2229] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:34.881165 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 6259
I20260926 06:57:34.881268 6424 raft_consensus.cc:2258] T 5f9bd24b9bf8467c940929d161c36791 P aa15cb94e2e347e5a4c6ece756c0c98e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:34.886688 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 5906
I20260926 06:57:34.886813 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:34.886679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:45945 (local address 127.0.0.1:53226)
0926 06:57:34.886740 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:34.886768 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.67:45945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
2026-09-26T06:57:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (571 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 06:57:34.906524 1321 mini_oidc.cc:152] Starting JWKS server
I20260926 06:57:34.907667 1321 webserver.cc:460] Webserver started at https://127.0.0.1:34095/ using document root <none> and password file <none>
I20260926 06:57:34.907788 1321 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 06:57:34.907974 1321 webserver.cc:460] Webserver started at http://127.0.0.1:33113/ using document root <none> and password file <none>
2026-09-26T06:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:34Z Disabled control of system clock
I20260926 06:57:34.915498 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:32797
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:32797
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34095/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/testchainca.cert with env {}
W20260926 06:57:34.995049 6432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:34.995249 6432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:34.995288 6432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:34.996393 6432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:34.996429 6432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:34.996443 6432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:34.996454 6432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:34.996467 6432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:57:34.996479 6432 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:57:34.996491 6432 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34095/jwks.json
I20260926 06:57:34.997905 6432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:32797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:32797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34095/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:34.998087 6432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:34.998250 6432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:57:35.003654 6432 server_base.cc:1034] running on GCE node
W20260926 06:57:35.003602 6437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:35.003602 6438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:35.003594 6440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:35.004098 6432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:35.004408 6432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:35.005574 6432 hybrid_clock.cc:648] HybridClock initialized: now 1790405855005576 us; error 43 us; skew 500 ppm
I20260926 06:57:35.006850 6432 webserver.cc:460] Webserver started at http://127.1.74.126:44879/ using document root <none> and password file <none>
I20260926 06:57:35.007102 6432 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:35.007174 6432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:35.007342 6432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:35.008421 6432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "da4e1cccf5894d63b30129626b1ef227"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "bd99a69df2bc2a43afa8b5039bf8340f"
server_key_iv: "baf904dd61166b7b26690810a0a17c7c"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.008780 6432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "da4e1cccf5894d63b30129626b1ef227"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "bd99a69df2bc2a43afa8b5039bf8340f"
server_key_iv: "baf904dd61166b7b26690810a0a17c7c"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.010088 6432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:35.010782 6446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:35.010957 6432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:35.011042 6432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "da4e1cccf5894d63b30129626b1ef227"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "bd99a69df2bc2a43afa8b5039bf8340f"
server_key_iv: "baf904dd61166b7b26690810a0a17c7c"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.011123 6432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:35.024489 6432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:35.024796 6432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:35.024981 6432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:35.028751 6432 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:32797
I20260926 06:57:35.028785 6498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:32797 every 8 connection(s)
I20260926 06:57:35.029116 6432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:35.029492 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 6432
I20260926 06:57:35.029594 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:35.029781 1321 external_mini_cluster.cc:1428] Setting key 97b38cb7d896006985829f29b1d21e25
I20260926 06:57:35.030278 6499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:35.032836 6499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227: Bootstrap starting.
I20260926 06:57:35.033416 6499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:35.033692 6499 log.cc:824] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:35.034356 6499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227: No bootstrap required, opened a new log
I20260926 06:57:35.035027 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:35.030298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.1.74.126:32797)
0926 06:57:35.030498 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:35.030506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:35.030530 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:57:35.030937 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:35.030943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:35.031000 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:35.031145 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:35.031336 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.032249 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.033113 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.033573 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.033704 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.034404 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:35.034408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:35.034408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:35.034594 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:35.034625 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:35.034635 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:35.034710 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:35.034781 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 06:57:35.034856 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260926 06:57:35.036151 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:57:35.036753 6499 raft_consensus.cc:348] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4e1cccf5894d63b30129626b1ef227" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 32797 } }
I20260926 06:57:35.036922 6499 raft_consensus.cc:374] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:35.036958 6499 raft_consensus.cc:729] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4e1cccf5894d63b30129626b1ef227, State: Initialized, Role: FOLLOWER
I20260926 06:57:35.037079 6499 consensus_queue.cc:260] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4e1cccf5894d63b30129626b1ef227" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 32797 } }
I20260926 06:57:35.037153 6499 raft_consensus.cc:388] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:35.037174 6499 raft_consensus.cc:482] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:35.037232 6499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:35.037741 6499 raft_consensus.cc:504] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4e1cccf5894d63b30129626b1ef227" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 32797 } }
I20260926 06:57:35.037858 6499 leader_election.cc:302] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da4e1cccf5894d63b30129626b1ef227; no voters:
I20260926 06:57:35.038008 6499 leader_election.cc:288] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:35.038091 6504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:35.038236 6504 raft_consensus.cc:686] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [term 1 LEADER]: Becoming Leader. State: Replica: da4e1cccf5894d63b30129626b1ef227, State: Running, Role: LEADER
I20260926 06:57:35.038264 6499 sys_catalog.cc:564] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:35.038317 6504 consensus_queue.cc:237] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4e1cccf5894d63b30129626b1ef227" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 32797 } }
I20260926 06:57:35.038623 6506 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da4e1cccf5894d63b30129626b1ef227. Latest consensus state: current_term: 1 leader_uuid: "da4e1cccf5894d63b30129626b1ef227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4e1cccf5894d63b30129626b1ef227" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 32797 } } }
I20260926 06:57:35.038630 6505 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da4e1cccf5894d63b30129626b1ef227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4e1cccf5894d63b30129626b1ef227" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 32797 } } }
I20260926 06:57:35.038769 6506 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:35.038806 6505 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227 [sys.catalog]: This master's current role is: LEADER
W20260926 06:57:35.040108 6520 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:35.040169 6520 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:35.040239 6511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:35.040392 6511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:35.041947 6511 catalog_manager.cc:1347] Generated new cluster ID: c9b9bbb9ed704d9cb30149b42578cb47
I20260926 06:57:35.041987 6511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:35.066780 6511 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:35.067324 6511 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:35.073999 6511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da4e1cccf5894d63b30129626b1ef227: Generated new TSK 0
I20260926 06:57:35.074180 6511 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 06:57:35.108198 6529 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 06:57:35.108417 6529 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 06:57:35.108533 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:35.086305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.1.74.126:32797)
0926 06:57:35.086368 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:35.086370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:35.086507 (+ 137us) server_negotiation.cc:407] Connection header received
0926 06:57:35.086606 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:35.086607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:35.086618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:35.086676 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:57:35.086845 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.087379 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.088508 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.088726 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.088773 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.089002 (+ 229us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:57:35.108252 (+ 19250us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 06:57:35.108458 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 06:57:35.108620 6502 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 06:57:35.108894 6528 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:35.086237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:32797 (local address 127.0.0.1:49602)
0926 06:57:35.086432 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:35.086443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:35.086516 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:35.086703 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:35.086704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:35.086706 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:57:35.086779 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:35.086781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.087467 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.087468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:35.088421 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:35.088424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.088793 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.088794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:35.088878 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.088900 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:57:35.108451 (+ 19551us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 06:57:35.108575 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.74.126:32797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":115,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.74.126:32797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34095/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 06:57:35.109282 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 6432
2026-09-26T06:57:35Z chronyd exiting
I20260926 06:57:35.145087 1321 test_util.cc:175] -----------------------------------------------
I20260926 06:57:35.145196 1321 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790405833506994-1321-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (239 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 06:57:35.145925 1321 mini_oidc.cc:152] Starting JWKS server
I20260926 06:57:35.146625 1321 webserver.cc:460] Webserver started at https://127.0.0.1:42097/ using document root <none> and password file <none>
I20260926 06:57:35.146693 1321 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 06:57:35.146873 1321 webserver.cc:460] Webserver started at http://127.0.0.1:35141/ using document root <none> and password file <none>
2026-09-26T06:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:35Z Disabled control of system clock
I20260926 06:57:35.154312 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46735
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:46735
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/testchainca.cert with env {}
W20260926 06:57:35.236198 6535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:35.236405 6535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:35.236428 6535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:35.237532 6535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:35.237568 6535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:35.237581 6535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:35.237593 6535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:35.237607 6535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:57:35.237618 6535 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:57:35.237629 6535 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42097/jwks.json
I20260926 06:57:35.239168 6535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:46735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:35.239429 6535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:35.239609 6535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:35.244618 6543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:35.244786 6535 server_base.cc:1034] running on GCE node
W20260926 06:57:35.244894 6540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:35.244622 6541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:35.245162 6535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:35.245472 6535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:35.246681 6535 hybrid_clock.cc:648] HybridClock initialized: now 1790405855246654 us; error 38 us; skew 500 ppm
I20260926 06:57:35.248165 6535 webserver.cc:460] Webserver started at http://127.1.74.126:39979/ using document root <none> and password file <none>
I20260926 06:57:35.248464 6535 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:35.248540 6535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:35.248692 6535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:35.249773 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "cad1889e6866415883024a9697714175"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "821a99cdf7121ed2d83847007d090870"
server_key_iv: "5f1fa2f2da5e7b0861d325d91cd8d320"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.250136 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "cad1889e6866415883024a9697714175"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "821a99cdf7121ed2d83847007d090870"
server_key_iv: "5f1fa2f2da5e7b0861d325d91cd8d320"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.251559 6535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:35.252290 6549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:35.252455 6535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:35.252558 6535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "cad1889e6866415883024a9697714175"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "821a99cdf7121ed2d83847007d090870"
server_key_iv: "5f1fa2f2da5e7b0861d325d91cd8d320"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.252632 6535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:35.263355 6535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:35.263620 6535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:35.263772 6535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:35.267400 6535 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:46735
I20260926 06:57:35.267449 6601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:46735 every 8 connection(s)
I20260926 06:57:35.267808 6535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:35.268479 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 6535
I20260926 06:57:35.268589 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:35.268808 1321 external_mini_cluster.cc:1428] Setting key a830b3e7dd3834f8f2126d2a5723225a
I20260926 06:57:35.268903 6602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:35.272189 6602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175: Bootstrap starting.
I20260926 06:57:35.272827 6602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:35.273113 6602 log.cc:824] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:35.273808 6602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175: No bootstrap required, opened a new log
I20260926 06:57:35.274485 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:35.269347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.1.74.126:46735)
0926 06:57:35.269541 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:35.269548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:35.269572 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:57:35.270028 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:35.270035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:35.270093 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:35.270210 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:35.270448 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.271362 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.272450 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.272920 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.273071 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.273803 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:35.273807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:35.273807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:35.274029 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:35.274061 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:35.274072 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:35.274143 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:35.274234 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 06:57:35.274314 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 06:57:35.275525 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:57:35.276161 6602 raft_consensus.cc:348] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad1889e6866415883024a9697714175" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46735 } }
I20260926 06:57:35.276361 6602 raft_consensus.cc:374] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:35.276422 6602 raft_consensus.cc:729] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad1889e6866415883024a9697714175, State: Initialized, Role: FOLLOWER
I20260926 06:57:35.276544 6602 consensus_queue.cc:260] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad1889e6866415883024a9697714175" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46735 } }
I20260926 06:57:35.276623 6602 raft_consensus.cc:388] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:35.276643 6602 raft_consensus.cc:482] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:35.276710 6602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:35.277240 6602 raft_consensus.cc:504] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad1889e6866415883024a9697714175" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46735 } }
I20260926 06:57:35.277325 6602 leader_election.cc:302] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cad1889e6866415883024a9697714175; no voters:
I20260926 06:57:35.277500 6602 leader_election.cc:288] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:35.277536 6607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:35.277707 6607 raft_consensus.cc:686] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [term 1 LEADER]: Becoming Leader. State: Replica: cad1889e6866415883024a9697714175, State: Running, Role: LEADER
I20260926 06:57:35.277752 6602 sys_catalog.cc:564] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:35.277797 6607 consensus_queue.cc:237] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad1889e6866415883024a9697714175" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46735 } }
I20260926 06:57:35.278107 6607 sys_catalog.cc:455] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cad1889e6866415883024a9697714175. Latest consensus state: current_term: 1 leader_uuid: "cad1889e6866415883024a9697714175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad1889e6866415883024a9697714175" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46735 } } }
I20260926 06:57:35.278182 6607 sys_catalog.cc:458] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:35.278231 6608 sys_catalog.cc:455] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cad1889e6866415883024a9697714175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad1889e6866415883024a9697714175" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46735 } } }
I20260926 06:57:35.278287 6608 sys_catalog.cc:458] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:35.279037 6612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:35.279544 6612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:35.281172 6612 catalog_manager.cc:1347] Generated new cluster ID: ae491edabb014dd5b3e1c10fb42f755c
I20260926 06:57:35.281237 6612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:35.305637 6612 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:35.306293 6612 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:35.310030 6612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cad1889e6866415883024a9697714175: Generated new TSK 0
I20260926 06:57:35.310251 6612 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 06:57:35.331378 6632 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 06:57:35.331466 6632 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 06:57:35.331827 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:35.316992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.1.74.126:46735)
0926 06:57:35.317037 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:35.317039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:35.317140 (+ 101us) server_negotiation.cc:407] Connection header received
0926 06:57:35.317207 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:35.317207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:35.317218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:35.317281 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:57:35.317434 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.317980 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.319136 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.319410 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.319465 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.319660 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:57:35.331640 (+ 11980us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 06:57:35.331776 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 06:57:35.331902 6605 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40946: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 06:57:35.332023 6631 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:35.316886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:46735 (local address 127.0.0.1:40946)
0926 06:57:35.317058 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:35.317070 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:35.317138 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:35.317306 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:35.317307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:35.317309 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:57:35.317367 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:35.317369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.318085 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.318086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:35.319049 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:35.319052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.319484 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.319486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:35.319555 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.319562 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:57:35.331768 (+ 12206us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 06:57:35.331923 (+ 155us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.74.126:46735: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260926 06:57:35.332392 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 6535
2026-09-26T06:57:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (223 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 06:57:35.369961 1321 mini_oidc.cc:152] Starting JWKS server
I20260926 06:57:35.370865 1321 webserver.cc:460] Webserver started at https://127.0.0.1:34907/ using document root <none> and password file <none>
I20260926 06:57:35.370941 1321 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 06:57:35.371115 1321 webserver.cc:460] Webserver started at http://127.0.0.1:44237/ using document root <none> and password file <none>
2026-09-26T06:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:35Z Disabled control of system clock
I20260926 06:57:35.378947 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:35189
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:38031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:35189
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34907/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 06:57:35.457958 6638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:35.458159 6638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:35.458180 6638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:35.459267 6638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:35.459303 6638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:35.459316 6638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:35.459328 6638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:35.459340 6638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:57:35.459353 6638 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:57:35.459364 6638 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34907/jwks.json
I20260926 06:57:35.460640 6638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:38031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:35189
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:35189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34907/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:35.460824 6638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:35.461050 6638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:35.465894 6643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:35.465894 6646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:35.465903 6644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:35.466229 6638 server_base.cc:1034] running on GCE node
I20260926 06:57:35.466450 6638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:35.466732 6638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:35.467883 6638 hybrid_clock.cc:648] HybridClock initialized: now 1790405855467870 us; error 31 us; skew 500 ppm
I20260926 06:57:35.469144 6638 webserver.cc:460] Webserver started at http://127.1.74.126:37735/ using document root <none> and password file <none>
I20260926 06:57:35.469341 6638 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:35.469384 6638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:35.469473 6638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:35.470583 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "3e16923e6aef47ffa28134eed60f3bd2"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "c6daa379d0e773627c7ddc016d910a98"
server_key_iv: "51853ec33d10b707fce2ec41f544f7c5"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.470950 6638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "3e16923e6aef47ffa28134eed60f3bd2"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "c6daa379d0e773627c7ddc016d910a98"
server_key_iv: "51853ec33d10b707fce2ec41f544f7c5"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.472306 6638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:35.473122 6652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:35.473296 6638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:35.473380 6638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "3e16923e6aef47ffa28134eed60f3bd2"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "c6daa379d0e773627c7ddc016d910a98"
server_key_iv: "51853ec33d10b707fce2ec41f544f7c5"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.473439 6638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:35.481807 6638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:35.482033 6638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:35.482128 6638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:35.485657 6638 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:35189
I20260926 06:57:35.485739 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:35189 every 8 connection(s)
I20260926 06:57:35.486037 6638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:35.486675 6705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:35.489226 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2: Bootstrap starting.
I20260926 06:57:35.489816 6705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:35.490097 6705 log.cc:824] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:35.490741 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2: No bootstrap required, opened a new log
I20260926 06:57:35.492359 6705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e16923e6aef47ffa28134eed60f3bd2" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35189 } }
I20260926 06:57:35.492515 6705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:35.492558 6705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e16923e6aef47ffa28134eed60f3bd2, State: Initialized, Role: FOLLOWER
I20260926 06:57:35.492740 6705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e16923e6aef47ffa28134eed60f3bd2" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35189 } }
I20260926 06:57:35.492862 6705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:35.492904 6705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:35.492947 6705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:35.493054 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 6638
I20260926 06:57:35.493141 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:35.493315 1321 external_mini_cluster.cc:1428] Setting key ecf08953facd59485657f62b47bb20b2
I20260926 06:57:35.493609 6705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e16923e6aef47ffa28134eed60f3bd2" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35189 } }
I20260926 06:57:35.493721 6705 leader_election.cc:302] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e16923e6aef47ffa28134eed60f3bd2; no voters:
I20260926 06:57:35.493935 6705 leader_election.cc:288] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:35.493983 6710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:35.494107 6710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [term 1 LEADER]: Becoming Leader. State: Replica: 3e16923e6aef47ffa28134eed60f3bd2, State: Running, Role: LEADER
I20260926 06:57:35.494211 6705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:35.494254 6710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e16923e6aef47ffa28134eed60f3bd2" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35189 } }
I20260926 06:57:35.494858 6711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e16923e6aef47ffa28134eed60f3bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e16923e6aef47ffa28134eed60f3bd2" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35189 } } }
I20260926 06:57:35.494938 6711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:35.495357 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e16923e6aef47ffa28134eed60f3bd2. Latest consensus state: current_term: 1 leader_uuid: "3e16923e6aef47ffa28134eed60f3bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e16923e6aef47ffa28134eed60f3bd2" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35189 } } }
I20260926 06:57:35.495441 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2 [sys.catalog]: This master's current role is: LEADER
W20260926 06:57:35.496011 6726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:35.496079 6726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:35.496173 6719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:35.496349 6719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:35.497754 6719 catalog_manager.cc:1347] Generated new cluster ID: e095455f344747078b3f31418e879c2b
I20260926 06:57:35.497807 6719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:35.499190 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:35.493775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.1.74.126:35189)
0926 06:57:35.493998 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:35.494007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:35.494040 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:57:35.494160 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:35.494167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:35.494236 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:35.494609 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:35.494705 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.496185 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.497074 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.497662 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.497817 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.498496 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:35.498499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:35.498500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:35.498684 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:35.498720 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:35.498733 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:35.498797 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:35.498929 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 06:57:35.499001 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260926 06:57:35.500083 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:57:35.517359 6719 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:35.517897 6719 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:35.524565 6719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e16923e6aef47ffa28134eed60f3bd2: Generated new TSK 0
I20260926 06:57:35.524755 6719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:57:35.540436 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:35.537182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.1.74.126:35189)
0926 06:57:35.537253 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:35.537255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:35.537340 (+ 85us) server_negotiation.cc:407] Connection header received
0926 06:57:35.537426 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:35.537426 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:35.537439 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:35.537522 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:57:35.537653 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.538190 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.538964 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.539151 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.539232 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.539446 (+ 214us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:57:35.540203 (+ 757us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 06:57:35.540351 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 06:57:35.540494 6709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 06:57:35.540521 6734 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:35.537114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:35189 (local address 127.0.0.1:56196)
0926 06:57:35.537270 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:35.537280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:35.537354 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:35.537520 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:35.537521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:35.537522 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:57:35.537586 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:35.537588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.538279 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.538280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:35.538870 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:35.538873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.539235 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.539237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:35.539323 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.539328 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:57:35.540311 (+ 983us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 06:57:35.540399 (+ 88us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.74.126:35189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260926 06:57:35.540735 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 6638
E20260926 06:57:35.551555 6735 webserver.cc:533] Webserver: sslize failed:
E20260926 06:57:35.551616 6735 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T06:57:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (208 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T06:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:35Z Disabled control of system clock
I20260926 06:57:35.585178 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33451
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:34807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:33451
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/testchainca.cert with env {}
W20260926 06:57:35.664681 6739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:35.664893 6739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:35.664916 6739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:35.666044 6739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:35.666078 6739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:35.666090 6739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:35.666103 6739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:35.666115 6739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:57:35.666127 6739 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:57:35.666138 6739 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 06:57:35.667578 6739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:33451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:35.667776 6739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:35.667943 6739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:35.672847 6745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:35.672875 6739 server_base.cc:1034] running on GCE node
W20260926 06:57:35.672879 6747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:35.672832 6744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:35.673256 6739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:35.673511 6739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:35.674647 6739 hybrid_clock.cc:648] HybridClock initialized: now 1790405855674636 us; error 27 us; skew 500 ppm
I20260926 06:57:35.680612 6739 webserver.cc:460] Webserver started at http://127.1.74.126:42395/ using document root <none> and password file <none>
I20260926 06:57:35.680862 6739 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:35.680910 6739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:35.681080 6739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:35.682205 6739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "c3ef96a9b197488e9dbbc87fae5c0e87"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "98697eb7809d7592d8acb8c92a900c67"
server_key_iv: "1e1baa002f7d5557eb3e12186cd285c9"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.682567 6739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "c3ef96a9b197488e9dbbc87fae5c0e87"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "98697eb7809d7592d8acb8c92a900c67"
server_key_iv: "1e1baa002f7d5557eb3e12186cd285c9"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.683939 6739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:35.684703 6753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:35.684888 6739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:35.684971 6739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "c3ef96a9b197488e9dbbc87fae5c0e87"
format_stamp: "Formatted at 2026-09-26 06:57:35 on dist-test-slave-sfvt"
server_key: "98697eb7809d7592d8acb8c92a900c67"
server_key_iv: "1e1baa002f7d5557eb3e12186cd285c9"
server_key_version: "encryptionkey@0"
I20260926 06:57:35.685058 6739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:35.710777 6739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:35.711139 6739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:35.711323 6739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:35.715204 6739 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:33451
I20260926 06:57:35.715309 6805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:33451 every 8 connection(s)
I20260926 06:57:35.715679 6739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:35.716584 6806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:35.718887 6806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87: Bootstrap starting.
I20260926 06:57:35.719481 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 6739
I20260926 06:57:35.719547 6806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:35.719594 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:35.719805 1321 external_mini_cluster.cc:1428] Setting key b243549daab75fb8f28692e300ba264d
I20260926 06:57:35.719894 6806 log.cc:824] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:35.720728 6806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87: No bootstrap required, opened a new log
I20260926 06:57:35.723012 6806 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef96a9b197488e9dbbc87fae5c0e87" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33451 } }
I20260926 06:57:35.723188 6806 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:35.723222 6806 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ef96a9b197488e9dbbc87fae5c0e87, State: Initialized, Role: FOLLOWER
I20260926 06:57:35.723380 6806 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef96a9b197488e9dbbc87fae5c0e87" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33451 } }
I20260926 06:57:35.723476 6806 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:35.723547 6806 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:35.723627 6806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:35.724285 6806 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef96a9b197488e9dbbc87fae5c0e87" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33451 } }
I20260926 06:57:35.724401 6806 leader_election.cc:302] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3ef96a9b197488e9dbbc87fae5c0e87; no voters:
I20260926 06:57:35.724587 6806 leader_election.cc:288] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:35.724613 6811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:35.724784 6811 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [term 1 LEADER]: Becoming Leader. State: Replica: c3ef96a9b197488e9dbbc87fae5c0e87, State: Running, Role: LEADER
I20260926 06:57:35.724887 6806 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:35.724957 6811 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef96a9b197488e9dbbc87fae5c0e87" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33451 } }
I20260926 06:57:35.725625 6812 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3ef96a9b197488e9dbbc87fae5c0e87. Latest consensus state: current_term: 1 leader_uuid: "c3ef96a9b197488e9dbbc87fae5c0e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef96a9b197488e9dbbc87fae5c0e87" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33451 } } }
I20260926 06:57:35.725714 6812 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:35.725792 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:35.720308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.1.74.126:33451)
0926 06:57:35.720503 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:35.720510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:35.720538 (+ 28us) server_negotiation.cc:407] Connection header received
0926 06:57:35.721024 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:35.721031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:35.721096 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:35.721210 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:35.721463 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.722301 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.723511 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.724058 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.724155 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.724993 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:35.724997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:35.724997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:57:35.725204 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:35.725249 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:57:35.725265 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:35.725332 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:35.725446 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 06:57:35.725530 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260926 06:57:35.726112 6813 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3ef96a9b197488e9dbbc87fae5c0e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3ef96a9b197488e9dbbc87fae5c0e87" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33451 } } }
I20260926 06:57:35.726249 6813 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:35.726382 6820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:35.726732 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:57:35.727094 6820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:35.728581 6820 catalog_manager.cc:1347] Generated new cluster ID: 666769e4fe804ce795d70c0f66103af3
I20260926 06:57:35.728652 6820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:35.740214 6820 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:35.740760 6820 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:35.747814 6820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3ef96a9b197488e9dbbc87fae5c0e87: Generated new TSK 0
I20260926 06:57:35.748036 6820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:57:35.825841 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:35.763791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.1.74.126:33451)
0926 06:57:35.763846 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:35.763848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:35.763962 (+ 114us) server_negotiation.cc:407] Connection header received
0926 06:57:35.764008 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:35.764009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:35.764021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:35.764102 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:57:35.764222 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.765018 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.765775 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.766091 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.766163 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.766362 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:57:35.825523 (+ 59161us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 06:57:35.825750 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 06:57:35.825922 6810 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 06:57:35.825984 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:35.763724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33451 (local address 127.0.0.1:59884)
0926 06:57:35.763878 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:35.763896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:35.763959 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:35.764105 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:35.764106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:35.764107 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:57:35.764160 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:35.764162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.765104 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.765104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:35.765695 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:35.765697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:35.766193 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:35.766194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:35.766248 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:35.766253 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:57:35.825669 (+ 59416us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 06:57:35.825866 (+ 197us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.74.126:33451: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260926 06:57:35.826498 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 6739
2026-09-26T06:57:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (265 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:36 dist-test-slave-sfvt krb5kdc[6847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:36 dist-test-slave-sfvt krb5kdc[6847](info): set up 2 sockets
Sep 26 06:57:36 dist-test-slave-sfvt krb5kdc[6847](info): commencing operation
krb5kdc: starting...
W20260926 06:57:37.043671 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:37.162936 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:35559
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:35559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:37.245146 6860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:37.245390 6860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:37.245428 6860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:37.246579 6860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:37.246650 6860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:37.246675 6860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:37.246694 6860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:37.246724 6860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:37.248196 6860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:35559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:35559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:37.248476 6860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:37.248698 6860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:37.253690 6868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:37.253731 6865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:37.253762 6860 server_base.cc:1034] running on GCE node
W20260926 06:57:37.253731 6866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:37.254276 6860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:37.254545 6860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:37.255693 6860 hybrid_clock.cc:648] HybridClock initialized: now 1790405857255668 us; error 40 us; skew 500 ppm
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.258263 6860 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:37.258741 6860 webserver.cc:460] Webserver started at http://127.1.74.126:37677/ using document root <none> and password file <none>
I20260926 06:57:37.258970 6860 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:37.259014 6860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:37.259164 6860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:37.260273 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "c0b5241d95a74a8d9f31122eda195c50"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "ea0ae4b9589d12187e6b2b1fa9250d51"
server_key_iv: "4e7e8713f4ab6c9d1fcd3de94be8039d"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.260653 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "c0b5241d95a74a8d9f31122eda195c50"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "ea0ae4b9589d12187e6b2b1fa9250d51"
server_key_iv: "4e7e8713f4ab6c9d1fcd3de94be8039d"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.261934 6860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:57:37.262646 6875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.262814 6860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:37.262918 6860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "c0b5241d95a74a8d9f31122eda195c50"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "ea0ae4b9589d12187e6b2b1fa9250d51"
server_key_iv: "4e7e8713f4ab6c9d1fcd3de94be8039d"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.262991 6860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:37.290200 6860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:37.292639 6860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:37.292814 6860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:37.296849 6860 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:35559
I20260926 06:57:37.296885 6927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:35559 every 8 connection(s)
I20260926 06:57:37.297241 6860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:37.297451 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 6860
I20260926 06:57:37.297549 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:37.297767 1321 external_mini_cluster.cc:1428] Setting key c020ce9372b7383254410135830f277b
I20260926 06:57:37.298229 6928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:37.300952 6928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50: Bootstrap starting.
I20260926 06:57:37.301548 6928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:37.301824 6928 log.cc:824] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:37.302553 6928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50: No bootstrap required, opened a new log
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:37.304464 6928 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b5241d95a74a8d9f31122eda195c50" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35559 } }
I20260926 06:57:37.304618 6928 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:37.304666 6928 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b5241d95a74a8d9f31122eda195c50, State: Initialized, Role: FOLLOWER
I20260926 06:57:37.304818 6928 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b5241d95a74a8d9f31122eda195c50" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35559 } }
I20260926 06:57:37.304912 6928 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:37.304961 6928 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:37.305017 6928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:37.305763 6928 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b5241d95a74a8d9f31122eda195c50" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35559 } }
I20260926 06:57:37.305877 6928 leader_election.cc:302] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0b5241d95a74a8d9f31122eda195c50; no voters:
I20260926 06:57:37.306049 6928 leader_election.cc:288] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:37.306160 6933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:37.306344 6928 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:37.306350 6933 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [term 1 LEADER]: Becoming Leader. State: Replica: c0b5241d95a74a8d9f31122eda195c50, State: Running, Role: LEADER
I20260926 06:57:37.306527 6933 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b5241d95a74a8d9f31122eda195c50" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35559 } }
I20260926 06:57:37.306908 6934 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0b5241d95a74a8d9f31122eda195c50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b5241d95a74a8d9f31122eda195c50" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35559 } } }
I20260926 06:57:37.307134 6934 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:37.307052 6935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0b5241d95a74a8d9f31122eda195c50. Latest consensus state: current_term: 1 leader_uuid: "c0b5241d95a74a8d9f31122eda195c50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b5241d95a74a8d9f31122eda195c50" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35559 } } }
I20260926 06:57:37.307403 6935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:37.307579 6938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:37.308112 6938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:37.308485 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.298289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.1.74.126:35559)
0926 06:57:37.298479 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.298488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.298512 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:57:37.298930 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.298936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.298980 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.299145 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.299792 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.300692 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.301492 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.302000 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.302123 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.304478 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:37.304487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:37.304488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:37.304498 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:37.306494 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.306956 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.306959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.306960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.307068 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.307322 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.307323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.307323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.307723 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:37.307884 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:37.308032 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 06:57:37.308190 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260926 06:57:37.309676 6938 catalog_manager.cc:1347] Generated new cluster ID: d789ec8948ed43eba7c486e75af72984
I20260926 06:57:37.309716 6938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:37.318892 6938 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:37.319476 6938 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:37.323045 6938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0b5241d95a74a8d9f31122eda195c50: Generated new TSK 0
I20260926 06:57:37.323251 6938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:37.350833 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35559
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:37.432371 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:37.432593 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:37.432631 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:37.433746 6954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:37.433811 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:37.433853 6954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:37.435328 6954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:37.435587 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:37.435810 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:37.440804 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:37.440815 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:37.440815 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:37.440985 6954 server_base.cc:1034] running on GCE node
I20260926 06:57:37.441253 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:37.441498 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:37.442639 6954 hybrid_clock.cc:648] HybridClock initialized: now 1790405857442615 us; error 50 us; skew 500 ppm
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.445297 6954 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:37.445679 6954 webserver.cc:460] Webserver started at http://127.1.74.65:43933/ using document root <none> and password file <none>
I20260926 06:57:37.445868 6954 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:37.445909 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:37.445994 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:37.447005 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "6c304149812a4ab29d3f858f27403608"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "7dfa1d902e55422e430caf629f9bbb9c"
server_key_iv: "c327add93ecf5e3883bc4648b430463c"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.447399 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "6c304149812a4ab29d3f858f27403608"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "7dfa1d902e55422e430caf629f9bbb9c"
server_key_iv: "c327add93ecf5e3883bc4648b430463c"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.448669 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:37.449373 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.449599 6954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:37.449704 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "6c304149812a4ab29d3f858f27403608"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "7dfa1d902e55422e430caf629f9bbb9c"
server_key_iv: "c327add93ecf5e3883bc4648b430463c"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.449785 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:37.462318 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:37.464293 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:37.464460 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:37.464702 6954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:37.465022 6954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:37.465077 6954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.465118 6954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:37.465157 6954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.470957 6954 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:37903
I20260926 06:57:37.471062 7082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:37903 every 8 connection(s)
I20260926 06:57:37.471422 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:37.475888 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 6954
I20260926 06:57:37.475965 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:37.476145 1321 external_mini_cluster.cc:1428] Setting key 57d037ba047f680469268548b5b191b6
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:37.480926 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.472197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:34703 (local address 127.1.74.126:35559)
0926 06:57:37.472282 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.472284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.472423 (+ 139us) server_negotiation.cc:407] Connection header received
0926 06:57:37.473345 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.473347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.473357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.473406 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.473987 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.474473 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.475461 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.475659 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.475748 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.477983 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:37.477984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:37.477985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:37.477992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:37.479512 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.479952 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.479954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.479954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.480013 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.480278 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.480279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.480280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.480391 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:37.480466 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:37.480759 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 06:57:37.480838 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 06:57:37.481175 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.472143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:35559 (local address 127.1.74.65:34703)
0926 06:57:37.472296 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:37.472332 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:37.473243 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:37.473471 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:37.473474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:37.473724 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:37.473910 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:37.473913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.474562 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.474563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:37.475375 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:37.475377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.475757 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.475758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:37.475859 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.476435 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:37.476445 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:37.477883 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:37.479609 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:37.479616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:37.479619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:37.479890 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:37.480080 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:37.480081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:37.480081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:37.480200 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:37.480457 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:37.480463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:37.480603 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 06:57:37.480726 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 06:57:37.480881 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260926 06:57:37.481619 7083 heartbeater.cc:347] Connected to a master server at 127.1.74.126:35559
I20260926 06:57:37.481727 7083 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:37.481921 7083 heartbeater.cc:511] Master 127.1.74.126:35559 requested a full tablet report, sending...
I20260926 06:57:37.482422 6892 ts_manager.cc:194] Registered new tserver with Master: 6c304149812a4ab29d3f858f27403608 (127.1.74.65:37903)
I20260926 06:57:37.483034 6892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:34703
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:37.501592 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35559
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:37.581768 7088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:37.581979 7088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:37.582002 7088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:37.583156 7088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:37.583201 7088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:37.583227 7088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:37.584748 7088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:37.585003 7088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:37.585232 7088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:37.590204 7096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:37.590181 7093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:37.590178 7094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:37.590533 7088 server_base.cc:1034] running on GCE node
I20260926 06:57:37.590807 7088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:37.591056 7088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:37.592224 7088 hybrid_clock.cc:648] HybridClock initialized: now 1790405857592223 us; error 41 us; skew 500 ppm
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.594705 7088 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:37.595124 7088 webserver.cc:460] Webserver started at http://127.1.74.66:32859/ using document root <none> and password file <none>
I20260926 06:57:37.595425 7088 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:37.595495 7088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:37.595654 7088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:37.596693 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "94dd0e27912742efa207f026ce3039f0"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "162b1c96e7f0f458fbf061a393ffecb4"
server_key_iv: "35a2437439848bec999558f4dccfa12e"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.597049 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "94dd0e27912742efa207f026ce3039f0"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "162b1c96e7f0f458fbf061a393ffecb4"
server_key_iv: "35a2437439848bec999558f4dccfa12e"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.598320 7088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:37.599012 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.599186 7088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:37.599298 7088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "94dd0e27912742efa207f026ce3039f0"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "162b1c96e7f0f458fbf061a393ffecb4"
server_key_iv: "35a2437439848bec999558f4dccfa12e"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.599385 7088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:37.631006 7088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:37.633416 7088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:37.633589 7088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:37.633836 7088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:37.634159 7088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:37.634213 7088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.634248 7088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:37.634274 7088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.640264 7088 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:42285
I20260926 06:57:37.640314 7216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:42285 every 8 connection(s)
I20260926 06:57:37.640659 7088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:37.646732 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 7088
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:37.646839 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:37.647006 1321 external_mini_cluster.cc:1428] Setting key 3c0136bccddade72d1da4b89b9d5c69e
I20260926 06:57:37.650197 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.641450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:56671 (local address 127.1.74.126:35559)
0926 06:57:37.641539 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.641541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.641644 (+ 103us) server_negotiation.cc:407] Connection header received
0926 06:57:37.642603 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.642604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.642615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.642682 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.643278 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.643822 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.644778 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.644972 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.645069 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.647356 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:37.647358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:37.647358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:37.647367 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:37.648781 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.649225 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.649227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.649227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.649288 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.649549 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.649550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.649550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.649661 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:37.649735 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:37.650010 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 06:57:37.650109 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 06:57:37.650400 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.641383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:35559 (local address 127.1.74.66:56671)
0926 06:57:37.641533 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:37.641575 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:37.642484 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:37.642755 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:37.642758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:37.643006 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:37.643194 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:37.643197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.643898 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.643899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:37.644698 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:37.644700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.645113 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.645114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:37.645254 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.645793 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:37.645801 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:37.647221 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:37.648906 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:37.648913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:37.648915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:37.649146 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:37.649355 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:37.649356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:37.649357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:37.649475 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:37.649727 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:37.649732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:37.649874 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 06:57:37.649991 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 06:57:37.650148 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 06:57:37.650959 7217 heartbeater.cc:347] Connected to a master server at 127.1.74.126:35559
I20260926 06:57:37.651084 7217 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:37.651295 7217 heartbeater.cc:511] Master 127.1.74.126:35559 requested a full tablet report, sending...
I20260926 06:57:37.651727 6892 ts_manager.cc:194] Registered new tserver with Master: 94dd0e27912742efa207f026ce3039f0 (127.1.74.66:42285)
I20260926 06:57:37.652292 6892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:56671
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:37.674353 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35559
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:37.753878 7222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:37.754115 7222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:37.754153 7222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:37.755338 7222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:37.755404 7222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:37.755448 7222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:37.756960 7222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:37.757232 7222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:37.757457 7222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:37.762490 7230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:37.762486 7228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:37.762487 7227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:37.762658 7222 server_base.cc:1034] running on GCE node
I20260926 06:57:37.762964 7222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:37.763211 7222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:37.764408 7222 hybrid_clock.cc:648] HybridClock initialized: now 1790405857764395 us; error 28 us; skew 500 ppm
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.766947 7222 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:37.767534 7222 webserver.cc:460] Webserver started at http://127.1.74.67:42253/ using document root <none> and password file <none>
I20260926 06:57:37.767797 7222 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:37.767848 7222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:37.767999 7222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:37.769078 7222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "7e0cfc0017e6418bbc9c9ed3ec8e837d"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "65bb570d078acc9d503b7593d5b05d1c"
server_key_iv: "7e1fcd2cf015686b059d27ab0eadddf4"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.769443 7222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "7e0cfc0017e6418bbc9c9ed3ec8e837d"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "65bb570d078acc9d503b7593d5b05d1c"
server_key_iv: "7e1fcd2cf015686b059d27ab0eadddf4"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.770753 7222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:37.771482 7237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.771665 7222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:37.771752 7222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "7e0cfc0017e6418bbc9c9ed3ec8e837d"
format_stamp: "Formatted at 2026-09-26 06:57:37 on dist-test-slave-sfvt"
server_key: "65bb570d078acc9d503b7593d5b05d1c"
server_key_iv: "7e1fcd2cf015686b059d27ab0eadddf4"
server_key_version: "encryptionkey@0"
I20260926 06:57:37.771824 7222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:37.781840 7222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:37.783630 7222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:37.783779 7222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:37.784016 7222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:37.784320 7222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:37.784354 7222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.784375 7222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:37.784423 7222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:37.790330 7222 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:40935
I20260926 06:57:37.790377 7350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:40935 every 8 connection(s)
I20260926 06:57:37.790688 7222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:37.799017 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 7222
I20260926 06:57:37.799124 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:37.799355 1321 external_mini_cluster.cc:1428] Setting key 4f917d272da0e6b77a115fb9ff9a7736
I20260926 06:57:37.800247 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.791462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:39807 (local address 127.1.74.126:35559)
0926 06:57:37.791532 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.791534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.791740 (+ 206us) server_negotiation.cc:407] Connection header received
0926 06:57:37.792797 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.792798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.792811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.792855 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.793479 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.794069 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.795143 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.795395 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.795478 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.797598 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:37.797599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:37.797599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:37.797606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:37.798777 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.799207 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.799208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.799209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.799324 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.799611 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.799612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.799612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.799733 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:37.799810 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:37.800085 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 06:57:37.800160 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:57:37.800433 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.791453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:35559 (local address 127.1.74.67:39807)
0926 06:57:37.791625 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:37.791660 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:37.792682 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:37.792921 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:37.792925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:37.793190 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:37.793387 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:37.793391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.794159 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.794161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:37.794982 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:37.794984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.795487 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.795488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:37.795667 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.796129 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:37.796137 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:37.797483 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:37.798887 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:37.798893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:37.798895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:37.799146 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:37.799399 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:37.799400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:37.799401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:37.799541 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:37.799800 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:37.799806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:37.799940 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 06:57:37.800054 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 06:57:37.800211 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260926 06:57:37.800868 7351 heartbeater.cc:347] Connected to a master server at 127.1.74.126:35559
I20260926 06:57:37.800953 7351 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:37.801091 7351 heartbeater.cc:511] Master 127.1.74.126:35559 requested a full tablet report, sending...
I20260926 06:57:37.801406 6892 ts_manager.cc:194] Registered new tserver with Master: 7e0cfc0017e6418bbc9c9ed3ec8e837d (127.1.74.67:40935)
I20260926 06:57:37.801781 6892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:39807
I20260926 06:57:37.811028 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/test-admin.keytab.
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.827080 1321 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 06:57:37.828694 7358 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:37.827945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.1.74.65:37903)
0926 06:57:37.828108 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.828113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.828123 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:37.828148 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.828150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.828164 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.828228 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.828596 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38938: BlockingRecv error: recv got EOF from 127.0.0.1:38938 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
W20260926 06:57:37.828754 1321 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5cc' of type FILE
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.829841 1321 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.830703 1321 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:37.838310 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.831198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38946 (local address 127.1.74.65:37903)
0926 06:57:37.831295 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.831297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.831308 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:57:37.831386 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.831386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.831396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.831446 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.831819 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.832547 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.833314 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.833687 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.833753 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.835917 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:37.835926 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:37.835926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:37.835933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:37.837346 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.837666 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.837669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.837669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.837725 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.837886 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.837887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.837887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.838010 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:37.838119 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:37.838178 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 06:57:37.838236 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 06:57:37.840250 7358 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:37.839616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.1.74.65:37903)
0926 06:57:37.839673 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.839675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.839712 (+ 37us) server_negotiation.cc:407] Connection header received
0926 06:57:37.839783 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.839784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.839794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.839866 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.840204 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38952: BlockingRecv error: recv got EOF from 127.0.0.1:38952 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 06:57:37.840330 1321 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5cc' of type FILE
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.841305 1321 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.842168 1321 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:37.849270 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.842632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.1.74.65:37903)
0926 06:57:37.842691 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.842693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.842747 (+ 54us) server_negotiation.cc:407] Connection header received
0926 06:57:37.842789 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.842790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.842798 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.842843 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.843176 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.843707 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.844346 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.844530 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.844578 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.846726 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:37.846728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:37.846728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:37.846735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:37.848165 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.848518 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.848519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.848520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.848576 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.848771 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.848772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.848772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.848869 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:37.848964 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:37.849095 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 06:57:37.849173 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 06:57:37.850809 7358 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:37.850246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.1.74.65:37903)
0926 06:57:37.850312 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.850314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.850322 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:37.850342 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.850343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.850352 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.850413 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.850758 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38976: BlockingRecv error: recv got EOF from 127.0.0.1:38976 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 06:57:37.851048 1321 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790405833506994-1321-0/krb5kdc/krb5cc' of type FILE
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.852083 1321 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:37.852882 1321 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[6847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:57:37.859715 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:37.853415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.1.74.65:37903)
0926 06:57:37.853472 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:37.853474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:37.853570 (+ 96us) server_negotiation.cc:407] Connection header received
0926 06:57:37.853642 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:37.853644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:37.853652 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:37.853695 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:37.854062 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.854528 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.855219 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:37.855460 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:37.855544 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:37.857436 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:37.857437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:37.857438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:37.857445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:37.858582 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.858926 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.858927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.858927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.858983 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:37.859181 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:37.859182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:37.859183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:37.859335 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:37.859443 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:37.859550 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 06:57:37.859621 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 06:57:37.860355 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 6954
I20260926 06:57:37.865199 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 7088
I20260926 06:57:37.869743 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 7222
I20260926 06:57:37.873883 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 6860
2026-09-26T06:57:37Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1872 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[7371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[7371](info): set up 2 sockets
Sep 26 06:57:37 dist-test-slave-sfvt krb5kdc[7371](info): commencing operation
krb5kdc: starting...
W20260926 06:57:38.912899 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:38 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.126.keytab.
Entry for principal oryx/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.126.keytab.
I20260926 06:57:39.030687 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:44973
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.126.keytab
--principal=oryx/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:44973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:39.115777 7384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:39.116026 7384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:39.116065 7384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:39.117494 7384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:39.117561 7384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:39.117585 7384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:39.117617 7384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:39.117646 7384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:39.119129 7384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:44973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.126.keytab
--principal=oryx/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:44973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:39.119410 7384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:39.119637 7384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:39.124576 7389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:39.124576 7390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:39.124636 7384 server_base.cc:1034] running on GCE node
W20260926 06:57:39.124595 7392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:39.125155 7384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:39.125460 7384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:39.126637 7384 hybrid_clock.cc:648] HybridClock initialized: now 1790405859126616 us; error 39 us; skew 500 ppm
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:39.129199 7384 init.cc:377] Logged in from keytab as oryx/127.1.74.126@KRBTEST.COM (short username oryx)
I20260926 06:57:39.129650 7384 webserver.cc:460] Webserver started at http://127.1.74.126:43651/ using document root <none> and password file <none>
I20260926 06:57:39.129902 7384 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:39.129948 7384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:39.130113 7384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:39.131153 7384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "83aa8acabfbb4e82b2cb782759ac993e"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "16489b7f7080bd5031230a5c23abe103"
server_key_iv: "d3c1c511fafbcdbc90db5f34901a8758"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.131575 7384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "83aa8acabfbb4e82b2cb782759ac993e"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "16489b7f7080bd5031230a5c23abe103"
server_key_iv: "d3c1c511fafbcdbc90db5f34901a8758"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.133009 7384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:39.133821 7399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.134029 7384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:39.134104 7384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "83aa8acabfbb4e82b2cb782759ac993e"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "16489b7f7080bd5031230a5c23abe103"
server_key_iv: "d3c1c511fafbcdbc90db5f34901a8758"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.134195 7384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:39.145557 7384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:39.147389 7384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:39.147537 7384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:39.151540 7384 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:44973
I20260926 06:57:39.151593 7451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:44973 every 8 connection(s)
I20260926 06:57:39.151957 7384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:39.152900 7452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:39.155409 7452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: Bootstrap starting.
I20260926 06:57:39.155438 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 7384
I20260926 06:57:39.155539 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:39.155726 1321 external_mini_cluster.cc:1428] Setting key 3c62b1555aaa977a1b0920760981cb29
I20260926 06:57:39.156172 7452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:39.156598 7452 log.cc:824] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:39.157379 7452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: No bootstrap required, opened a new log
I20260926 06:57:39.159729 7452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } }
I20260926 06:57:39.159879 7452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:39.159902 7452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83aa8acabfbb4e82b2cb782759ac993e, State: Initialized, Role: FOLLOWER
I20260926 06:57:39.160046 7452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } }
I20260926 06:57:39.160138 7452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:39.160184 7452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:39.160243 7452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:39.160950 7452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } }
I20260926 06:57:39.161057 7452 leader_election.cc:302] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83aa8acabfbb4e82b2cb782759ac993e; no voters:
I20260926 06:57:39.161224 7452 leader_election.cc:288] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:39.161268 7457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:39.161520 7457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 1 LEADER]: Becoming Leader. State: Replica: 83aa8acabfbb4e82b2cb782759ac993e, State: Running, Role: LEADER
I20260926 06:57:39.161578 7452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:39.161687 7457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } }
I20260926 06:57:39.162048 7457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83aa8acabfbb4e82b2cb782759ac993e. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } }
I20260926 06:57:39.162060 7458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } }
I20260926 06:57:39.162205 7457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:39.162433 7458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
W20260926 06:57:39.163430 7473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:39.163491 7473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:39.163537 7466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:39.163771 7466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:39.165692 7466 catalog_manager.cc:1347] Generated new cluster ID: 7657cceb74fa4fb5b423da56bfabfe83
I20260926 06:57:39.165786 7466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:39.167944 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.156262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.1.74.126:44973)
0926 06:57:39.156523 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.156534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.156574 (+ 40us) server_negotiation.cc:407] Connection header received
0926 06:57:39.157189 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.157198 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.157263 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.157420 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.158040 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.159558 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.160539 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.161200 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.161318 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.164222 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.164227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.164227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.164240 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.166309 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.166713 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.166716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.166717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.166827 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.167038 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.167039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.167039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.167347 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.167481 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.167619 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.167743 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":154,"threads_started":1}
I20260926 06:57:39.171466 7466 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:39.172180 7466 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:39.176117 7466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: Generated new TSK 0
I20260926 06:57:39.176294 7466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab.
Entry for principal oryx/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab.
I20260926 06:57:39.209898 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:44973
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab
--principal=oryx/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:39.308387 7478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:39.308630 7478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:39.308667 7478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:39.309808 7478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:39.309881 7478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:39.309926 7478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:39.311393 7478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab
--principal=oryx/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:39.311652 7478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:39.311894 7478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:39.317000 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:39.317046 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:39.317291 7486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:39.317934 7478 server_base.cc:1034] running on GCE node
I20260926 06:57:39.318269 7478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:39.318594 7478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:39.319788 7478 hybrid_clock.cc:648] HybridClock initialized: now 1790405859319757 us; error 48 us; skew 500 ppm
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:39.322782 7478 init.cc:377] Logged in from keytab as oryx/127.1.74.65@KRBTEST.COM (short username oryx)
I20260926 06:57:39.323316 7478 webserver.cc:460] Webserver started at http://127.1.74.65:44925/ using document root <none> and password file <none>
I20260926 06:57:39.323562 7478 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:39.323644 7478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:39.323782 7478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:39.325184 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "0bcbe55244ab41129e2ebf22ccfdac68"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "544ad3dd3f1033ad29055c034fecccb7"
server_key_iv: "7552be73a4fea69100d05e71e949aaf2"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.325655 7478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "0bcbe55244ab41129e2ebf22ccfdac68"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "544ad3dd3f1033ad29055c034fecccb7"
server_key_iv: "7552be73a4fea69100d05e71e949aaf2"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.327181 7478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 06:57:39.328017 7493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.328205 7478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:39.328310 7478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "0bcbe55244ab41129e2ebf22ccfdac68"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "544ad3dd3f1033ad29055c034fecccb7"
server_key_iv: "7552be73a4fea69100d05e71e949aaf2"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.328393 7478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:39.340082 7478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:39.342542 7478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:39.342721 7478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:39.342973 7478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:39.343365 7478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:39.343406 7478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.343448 7478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:39.343478 7478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.349634 7478 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:44673
I20260926 06:57:39.349762 7606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:44673 every 8 connection(s)
I20260926 06:57:39.350060 7478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:39.355496 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 7478
I20260926 06:57:39.355583 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:39.355791 1321 external_mini_cluster.cc:1428] Setting key 7e60f9f7153a1987032f762965c6e69d
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:39.360179 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.350842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:35923 (local address 127.1.74.126:44973)
0926 06:57:39.350905 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.350907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.351117 (+ 210us) server_negotiation.cc:407] Connection header received
0926 06:57:39.351677 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.351678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.351687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.351758 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.352692 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.353220 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.354265 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.354491 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.354555 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.356976 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.356978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.356978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.356986 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.358633 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.359115 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.359116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.359116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.359182 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.359476 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.359477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.359478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.359607 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.359710 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.359991 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.360085 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:57:39.360374 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.350810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.65:35923)
0926 06:57:39.350984 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:39.351026 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:39.351581 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:39.351814 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:39.351817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:39.352381 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:39.352573 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.352578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.353345 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.353346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.354163 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.354166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.354579 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.354580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.354693 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.355321 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:39.355332 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:39.356855 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:39.358742 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.358750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.358752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.359013 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.359285 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.359285 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.359286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.359398 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.359703 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:39.359710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:39.359848 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 06:57:39.359955 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 06:57:39.360125 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260926 06:57:39.361070 7607 heartbeater.cc:347] Connected to a master server at 127.1.74.126:44973
I20260926 06:57:39.361195 7607 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:39.361413 7607 heartbeater.cc:511] Master 127.1.74.126:44973 requested a full tablet report, sending...
I20260926 06:57:39.361974 7416 ts_manager.cc:194] Registered new tserver with Master: 0bcbe55244ab41129e2ebf22ccfdac68 (127.1.74.65:44673)
I20260926 06:57:39.362586 7416 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:35923
WARNING: no policy specified for oryx/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab.
Entry for principal oryx/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab.
I20260926 06:57:39.382679 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:44973
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab
--principal=oryx/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:39.463985 7612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:39.464186 7612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:39.464210 7612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:39.465283 7612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:39.465317 7612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:39.465342 7612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:39.466747 7612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab
--principal=oryx/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:39.466935 7612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:39.467100 7612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:39.471968 7618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:39.472035 7620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:39.472070 7612 server_base.cc:1034] running on GCE node
W20260926 06:57:39.472035 7617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:39.472466 7612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:39.472704 7612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:39.473839 7612 hybrid_clock.cc:648] HybridClock initialized: now 1790405859473822 us; error 33 us; skew 500 ppm
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:39.476426 7612 init.cc:377] Logged in from keytab as oryx/127.1.74.66@KRBTEST.COM (short username oryx)
I20260926 06:57:39.476794 7612 webserver.cc:460] Webserver started at http://127.1.74.66:35155/ using document root <none> and password file <none>
I20260926 06:57:39.476982 7612 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:39.477022 7612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:39.477108 7612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:39.478058 7612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "c59a69a04ee3450289da29995b78644c"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "02320d4960b7801e4a768ab820ceafa6"
server_key_iv: "03086fcb121cee99822ffe15874f0fd5"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.478344 7612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "c59a69a04ee3450289da29995b78644c"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "02320d4960b7801e4a768ab820ceafa6"
server_key_iv: "03086fcb121cee99822ffe15874f0fd5"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.479609 7612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:57:39.480242 7627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.480393 7612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:39.480479 7612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "c59a69a04ee3450289da29995b78644c"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "02320d4960b7801e4a768ab820ceafa6"
server_key_iv: "03086fcb121cee99822ffe15874f0fd5"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.480525 7612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:39.498152 7612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:39.499994 7612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:39.500100 7612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:39.500272 7612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:39.500572 7612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:39.500609 7612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.500631 7612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:39.500680 7612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.506115 7612 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:42713
I20260926 06:57:39.506191 7740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:42713 every 8 connection(s)
I20260926 06:57:39.506482 7612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:39.507359 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 7612
I20260926 06:57:39.507459 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:39.507653 1321 external_mini_cluster.cc:1428] Setting key 281827634a9daa34605ca0920ae4858c
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.74.67@KRBTEST.COM; defaulting to no policy
I20260926 06:57:39.517031 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.507298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:46099 (local address 127.1.74.126:44973)
0926 06:57:39.507353 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.507356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.507497 (+ 141us) server_negotiation.cc:407] Connection header received
0926 06:57:39.508076 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.508077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.508085 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.508130 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.509124 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.509858 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.511457 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.511637 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.511692 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.514165 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.514167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.514167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.514176 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.515782 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.516151 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.516152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.516153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.516209 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.516439 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.516440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.516440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.516587 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.516662 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.516904 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.516965 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:57:39.517194 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.507160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.66:46099)
0926 06:57:39.507372 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:39.507414 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:39.507977 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:39.508207 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:39.508211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:39.508832 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:39.509024 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.509029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.509960 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.509961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.511373 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.511376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.511729 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.511730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.511884 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.512417 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:39.512428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:39.514069 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:39.515862 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.515867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.515869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.516075 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.516276 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.516277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.516277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.516369 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.516657 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:39.516661 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:39.516778 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 06:57:39.516863 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 06:57:39.516978 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":122,"threads_started":1}
I20260926 06:57:39.517737 7741 heartbeater.cc:347] Connected to a master server at 127.1.74.126:44973
I20260926 06:57:39.517839 7741 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:39.518016 7741 heartbeater.cc:511] Master 127.1.74.126:44973 requested a full tablet report, sending...
I20260926 06:57:39.518399 7416 ts_manager.cc:194] Registered new tserver with Master: c59a69a04ee3450289da29995b78644c (127.1.74.66:42713)
I20260926 06:57:39.518790 7416 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:46099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab.
Entry for principal oryx/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab.
I20260926 06:57:39.535629 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:44973
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab
--principal=oryx/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:39.615490 7746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:39.615691 7746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:39.615715 7746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:39.616884 7746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:39.616928 7746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:39.616954 7746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:39.618386 7746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab
--principal=oryx/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:39.618634 7746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:39.618891 7746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:39.623898 7752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:39.623930 7751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:39.623905 7754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:39.624143 7746 server_base.cc:1034] running on GCE node
I20260926 06:57:39.624354 7746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:39.624579 7746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:39.625742 7746 hybrid_clock.cc:648] HybridClock initialized: now 1790405859625729 us; error 24 us; skew 500 ppm
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:39.628312 7746 init.cc:377] Logged in from keytab as oryx/127.1.74.67@KRBTEST.COM (short username oryx)
I20260926 06:57:39.628780 7746 webserver.cc:460] Webserver started at http://127.1.74.67:42791/ using document root <none> and password file <none>
I20260926 06:57:39.629014 7746 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:39.629081 7746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:39.629231 7746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:39.630266 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "376e271855ce4c63bc91e29cae1041da"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "d23fa5a52b780d917841282cd9cc24fa"
server_key_iv: "bf96dd86040867400559528ed1487067"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.630690 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "376e271855ce4c63bc91e29cae1041da"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "d23fa5a52b780d917841282cd9cc24fa"
server_key_iv: "bf96dd86040867400559528ed1487067"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.632189 7746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:39.632912 7761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.633090 7746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:39.633184 7746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "376e271855ce4c63bc91e29cae1041da"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "d23fa5a52b780d917841282cd9cc24fa"
server_key_iv: "bf96dd86040867400559528ed1487067"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.633270 7746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:39.642750 7746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:39.644724 7746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:39.644905 7746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:39.645119 7746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:39.645416 7746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:39.645470 7746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.645511 7746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:39.645539 7746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.651562 7746 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:39347
I20260926 06:57:39.651618 7874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:39347 every 8 connection(s)
I20260926 06:57:39.651924 7746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:39.660506 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 7746
I20260926 06:57:39.660646 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:39.660849 1321 external_mini_cluster.cc:1428] Setting key f8158f8f015227bb526b0206f3e60ed0
I20260926 06:57:39.661396 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.652656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:59455 (local address 127.1.74.126:44973)
0926 06:57:39.652710 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.652713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.652901 (+ 188us) server_negotiation.cc:407] Connection header received
0926 06:57:39.653398 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.653398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.653407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.653467 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.654425 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.654915 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.655910 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.656090 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.656149 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.658407 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.658409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.658409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.658416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.659846 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.660208 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.660209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.660209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.660256 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.660555 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.660556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.660556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.660706 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.660784 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.661126 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.661307 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:57:39.661576 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.652598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.67:59455)
0926 06:57:39.652768 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:39.652811 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:39.653314 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:39.653540 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:39.653543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:39.654109 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:39.654338 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.654343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.654991 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.654991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.655837 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.655839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.656163 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.656164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.656303 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.656805 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:39.656813 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:39.658293 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:39.659924 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.659930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.659932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.660146 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.660357 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.660358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.660358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.660483 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.660795 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:39.660801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:39.660960 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 06:57:39.661102 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 06:57:39.661326 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260926 06:57:39.662030 7875 heartbeater.cc:347] Connected to a master server at 127.1.74.126:44973
I20260926 06:57:39.662142 7875 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:39.662345 7875 heartbeater.cc:511] Master 127.1.74.126:44973 requested a full tablet report, sending...
I20260926 06:57:39.662741 7416 ts_manager.cc:194] Registered new tserver with Master: 376e271855ce4c63bc91e29cae1041da (127.1.74.67:39347)
I20260926 06:57:39.663268 7416 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:59455
I20260926 06:57:39.672765 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.74.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.125.keytab.
Entry for principal oryx/127.1.74.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.125.keytab.
I20260926 06:57:39.697912 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.125:40745
--webserver_interface=127.1.74.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.125.keytab
--principal=oryx/127.1.74.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:44973,127.1.74.125:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:39.778339 7880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:39.778541 7880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:39.778563 7880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:39.779760 7880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:39.779799 7880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:39.779814 7880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:39.779826 7880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:39.779838 7880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:39.781301 7880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:44973,127.1.74.125:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.125.keytab
--principal=oryx/127.1.74.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.125:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.74.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
W20260926 06:57:39.781394 7880 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.74.126:44973,127.1.74.125:40745'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 06:57:39.781553 7880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:39.781787 7880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:39.786465 7888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:39.786563 7886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:39.786782 7880 server_base.cc:1034] running on GCE node
W20260926 06:57:39.786818 7885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:39.787045 7880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:39.787351 7880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:39.788448 7880 hybrid_clock.cc:648] HybridClock initialized: now 1790405859788422 us; error 45 us; skew 500 ppm
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:39.792063 7880 init.cc:377] Logged in from keytab as oryx/127.1.74.125@KRBTEST.COM (short username oryx)
I20260926 06:57:39.792652 7880 webserver.cc:460] Webserver started at http://127.1.74.125:37995/ using document root <none> and password file <none>
I20260926 06:57:39.792901 7880 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:39.792982 7880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:39.793116 7880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:39.794634 7880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data/instance:
uuid: "c030d44f0c72420495cd8c27f9b508ae"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "bfa22d849ba6e4283224fec3635c9835"
server_key_iv: "5399fe429f1338817f22d3e0a9cbca98"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.795053 7880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal/instance:
uuid: "c030d44f0c72420495cd8c27f9b508ae"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "bfa22d849ba6e4283224fec3635c9835"
server_key_iv: "5399fe429f1338817f22d3e0a9cbca98"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.796846 7880 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 06:57:39.797737 7895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.797974 7880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:57:39.798074 7880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal
uuid: "c030d44f0c72420495cd8c27f9b508ae"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "bfa22d849ba6e4283224fec3635c9835"
server_key_iv: "5399fe429f1338817f22d3e0a9cbca98"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.798151 7880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:39.828462 7880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:39.831218 7880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:39.831393 7880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:39.836335 7880 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.125:40745
I20260926 06:57:39.836371 7947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.125:40745 every 8 connection(s)
I20260926 06:57:39.836740 7880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
I20260926 06:57:39.837816 7948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:39.839468 7948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:57:39.845662 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 7880
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.125@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:39.849968 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.840282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.1.74.126:44973)
0926 06:57:39.840333 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.840335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.840615 (+ 280us) server_negotiation.cc:407] Connection header received
0926 06:57:39.841424 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.841426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.841436 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.841486 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.842420 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.842951 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.844346 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.844535 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.844601 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.846900 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.846902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.846902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.846910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.848491 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.848915 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.848916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.848917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.848977 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.849228 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.849229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.849230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.849382 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.849460 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.849762 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.849852 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:57:39.850401 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.840285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34510)
0926 06:57:39.840456 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:39.840507 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:39.841241 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:39.841590 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:39.841595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:39.841986 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:39.842300 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.842306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.843091 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.843092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.844266 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.844268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.844613 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.844613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.844705 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.845197 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:39.845205 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:39.846796 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:39.848593 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.848599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.848601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.848824 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.849043 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.849044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.849044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.849158 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.849452 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:39.849458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:39.849597 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 06:57:39.849701 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 06:57:39.850139 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260926 06:57:39.851768 7948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.74.125@KRBTEST.COM
I20260926 06:57:39.856022 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.846203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.1.74.125:40745)
0926 06:57:39.846552 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.846557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.846579 (+ 22us) server_negotiation.cc:407] Connection header received
0926 06:57:39.846602 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.846604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.846632 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.846724 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.847605 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.848293 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.849084 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.849399 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.849497 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.852426 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.852433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.852434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.852440 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.854412 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.854835 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.854839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.854839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.854910 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.855198 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.855199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.855199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.855395 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.855564 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.855609 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.855676 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":101,"threads_started":1}
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.125@KRBTEST.COM for oryx/127.1.74.125@KRBTEST.COM
I20260926 06:57:39.859371 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.852054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.0.0.1:34982)
0926 06:57:39.852224 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:39.852234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:39.852270 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:39.852422 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:39.852423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:39.852610 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:39.852670 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.852672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.853492 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.853493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.854278 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.854281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.854747 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.854748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.854861 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.855360 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:39.855364 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:39.856733 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:39.858359 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.858360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.858361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.858525 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.858734 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.858734 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.858735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.858785 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.859033 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:39.859034 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:39.859070 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 06:57:39.859138 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 06:57:39.859234 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260926 06:57:39.859465 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.852107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.1.74.125:40745)
0926 06:57:39.852274 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.852276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.852310 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:57:39.852330 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.852330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.852340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.852403 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.852768 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.853383 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.854365 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.854632 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.854705 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.856824 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.856825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.856826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.856832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.858240 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.858607 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.858608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.858609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.858659 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.858855 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.858856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.858856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.858968 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.859041 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.859165 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.859308 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260926 06:57:39.860740 7948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Bootstrap starting.
I20260926 06:57:39.861402 7948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:39.861680 7948 log.cc:824] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:39.862546 7948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: No bootstrap required, opened a new log
I20260926 06:57:39.864039 7948 raft_consensus.cc:348] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } }
I20260926 06:57:39.864161 7948 raft_consensus.cc:374] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:39.864207 7948 raft_consensus.cc:729] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c030d44f0c72420495cd8c27f9b508ae, State: Initialized, Role: FOLLOWER
I20260926 06:57:39.864346 7948 consensus_queue.cc:260] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } }
I20260926 06:57:39.864646 7948 sys_catalog.cc:564] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:39.864604 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } } }
I20260926 06:57:39.864710 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:57:39.866315 7966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:39.866390 7966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:39.867973 7880 master_runner.cc:428] Detected that this master c030d44f0c72420495cd8c27f9b508ae is joining an existing cluster
I20260926 06:57:39.868026 7880 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.74.125:40745
I20260926 06:57:39.876156 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.869109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.1.74.126:44973)
0926 06:57:39.869157 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.869160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.869340 (+ 180us) server_negotiation.cc:407] Connection header received
0926 06:57:39.869467 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.869468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.869480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.869534 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.869909 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.870649 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.871738 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.872038 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.872099 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.873296 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.873298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.873298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.873305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.874911 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.875236 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.875237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.875238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.875330 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.875564 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.875565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.875565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.875693 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.875774 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.875962 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.876059 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:57:39.876308 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.869034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.0.0.1:34996)
0926 06:57:39.869260 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:39.869270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:39.869331 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:39.869513 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:39.869514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:39.869716 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:39.869779 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.869781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.871346 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.871347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.871998 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.872000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.872497 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.872498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.872565 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.872915 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:39.872919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:39.873275 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:39.875227 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.875228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.875228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.875420 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.875602 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.875603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.875603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.875671 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.875949 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:39.875950 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:39.875991 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 06:57:39.876063 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 06:57:39.876165 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"spinlock_wait_cycles":10112,"thread_start_us":168,"threads_started":1}
I20260926 06:57:39.876657 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.869191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.1.74.125:40745)
0926 06:57:39.869228 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.869230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.869336 (+ 106us) server_negotiation.cc:407] Connection header received
0926 06:57:39.869390 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.869391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.869400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.869476 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.869898 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.870649 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.872082 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.872416 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.872488 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.873364 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.873365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.873366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.873372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.874920 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.875499 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.875500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.875500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.875548 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.875739 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.875740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.875740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.875874 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.875949 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.876097 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.876225 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":20}
I20260926 06:57:39.876673 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.869034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34520)
0926 06:57:39.869270 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:39.869281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:39.869367 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:39.869549 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:39.869550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:39.869765 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:39.869831 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.869833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.870732 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.870733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.871655 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.871657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.872169 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.872169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.872212 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.872718 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:39.872722 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:39.873196 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:39.875000 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.875001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.875002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.875168 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.875409 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.875410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.875410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.875471 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.875801 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:39.875802 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:39.875843 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 06:57:39.876461 (+ 618us) client_negotiation.cc:240] Negotiation successful
0926 06:57:39.876553 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":4,"thread_start_us":99,"threads_started":1}
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.126@KRBTEST.COM for oryx/127.1.74.125@KRBTEST.COM
I20260926 06:57:39.885963 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.878738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.0.0.1:35006)
0926 06:57:39.878869 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:39.878913 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:39.878983 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:39.879115 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:39.879118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:39.879474 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:39.879528 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.879530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.880112 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.880113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.881065 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.881067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.881373 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.881374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.881452 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.881839 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:39.881844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:39.883334 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:39.884946 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.884952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.884953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.885169 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.885382 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.885382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.885382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.885437 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.885678 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:39.885683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:39.885722 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 06:57:39.885784 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:57:39.885869 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260926 06:57:39.885990 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.878818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.1.74.125:40745)
0926 06:57:39.878855 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.878857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.878967 (+ 110us) server_negotiation.cc:407] Connection header received
0926 06:57:39.879053 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.879053 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.879061 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.879101 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.879600 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.880038 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.881143 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.881306 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.881354 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.883407 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.883409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.883409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.883416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.884864 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.885239 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.885240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.885240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.885318 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.885495 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.885496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.885496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.885619 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.885683 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.885829 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.885901 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:57:39.886610 7416 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.74.125:40745: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae"
member_type: NON_VOTER
last_known_addr {
host: "127.1.74.125"
port: 40745
}
attrs {
promote: true
}
}
dest_uuid: "83aa8acabfbb4e82b2cb782759ac993e"
cas_config_opid_index: -1
I20260926 06:57:39.886930 7416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } }
I20260926 06:57:39.887401 7976 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c030d44f0c72420495cd8c27f9b508ae (127.1.74.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } } }
I20260926 06:57:39.887732 7922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:39.888511 7976 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.74.125:40745
I20260926 06:57:39.888622 7922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Refusing update from remote peer 83aa8acabfbb4e82b2cb782759ac993e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 06:57:39.888732 7880 master.cc:537] Master@127.1.74.125:40745 shutting down...
I20260926 06:57:39.888777 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } } }
I20260926 06:57:39.888849 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:39.889020 7976 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 06:57:39.889765 7400 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 06:57:39.890182 7400 consensus_peers.cc:487] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e -> Peer c030d44f0c72420495cd8c27f9b508ae (127.1.74.125:40745): Couldn't send request to peer c030d44f0c72420495cd8c27f9b508ae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 06:57:39.891547 7880 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:39.891637 7880 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:39.891709 7880 tablet_replica.cc:330] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: stopping tablet replica
I20260926 06:57:39.904575 7880 master.cc:552] Master@127.1.74.125:40745 shutdown complete.
I20260926 06:57:39.904637 7880 master_runner.cc:305] Clearing existing system tablet
I20260926 06:57:39.904778 7880 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:39.904837 7880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:39.905558 7880 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260926 06:57:39.906090 7980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.906240 7880 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:39.906301 7880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.906327 7880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal
uuid: "c030d44f0c72420495cd8c27f9b508ae"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "bfa22d849ba6e4283224fec3635c9835"
server_key_iv: "5399fe429f1338817f22d3e0a9cbca98"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.907114 7880 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:39.908893 7880 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 06:57:39.908963 7880 log.cc:1196] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 06:57:39.909150 7880 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Deleting consensus metadata
I20260926 06:57:39.909271 7880 master_runner.cc:315] Copying system tablet from 127.1.74.126:44973
I20260926 06:57:39.909880 7880 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: tablet copy: Beginning tablet copy session from remote peer at address 127.1.74.126:44973
I20260926 06:57:39.916126 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.910243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.1.74.126:44973)
0926 06:57:39.910284 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.910286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.910346 (+ 60us) server_negotiation.cc:407] Connection header received
0926 06:57:39.910430 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.910431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.910440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.910486 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.910817 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.911378 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.912436 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.912611 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.912659 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.913686 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.913688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.913688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.913697 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.915102 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.915487 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.915488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.915488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.915536 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.915713 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.915713 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.915714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.915803 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.915848 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.916009 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.916063 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:57:39.916203 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.910136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34522)
0926 06:57:39.910270 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:39.910285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:39.910363 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:39.910493 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:39.910493 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:39.910685 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:39.910751 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.910752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.911451 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.911452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.912354 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:39.912357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.912679 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.912680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:39.912748 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.913190 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:39.913195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:39.913575 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:39.915181 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.915182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.915182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.915417 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.915596 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:39.915596 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:39.915597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:39.915658 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:39.915859 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:39.915860 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:39.915907 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:57:39.915992 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 06:57:39.916094 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260926 06:57:39.916476 7436 tablet_copy_service.cc:140] P 83aa8acabfbb4e82b2cb782759ac993e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c030d44f0c72420495cd8c27f9b508ae ({username='oryx', principal='oryx/127.1.74.125@KRBTEST.COM'} at 127.0.0.1:34522)
I20260926 06:57:39.916574 7436 tablet_copy_service.cc:161] P 83aa8acabfbb4e82b2cb782759ac993e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c030d44f0c72420495cd8c27f9b508ae at {username='oryx', principal='oryx/127.1.74.125@KRBTEST.COM'} at 127.0.0.1:34522: session id = c030d44f0c72420495cd8c27f9b508ae-00000000000000000000000000000000
I20260926 06:57:39.917207 7436 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 06:57:39.917820 7880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 06:57:39.918728 7880 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: tablet copy: Starting download of 0 data blocks...
I20260926 06:57:39.918872 7880 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: tablet copy: Starting download of 1 WAL segments...
I20260926 06:57:39.919761 7880 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 06:57:39.920538 7436 tablet_copy_service.cc:342] P 83aa8acabfbb4e82b2cb782759ac993e: Request end of tablet copy session c030d44f0c72420495cd8c27f9b508ae-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.74.125@KRBTEST.COM'} at 127.0.0.1:34522
I20260926 06:57:39.920594 7436 tablet_copy_service.cc:433] P 83aa8acabfbb4e82b2cb782759ac993e: ending tablet copy session c030d44f0c72420495cd8c27f9b508ae-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c030d44f0c72420495cd8c27f9b508ae
I20260926 06:57:39.923113 7880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:39.924499 7987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:39.924507 7990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:39.924553 7988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:39.927425 7880 server_base.cc:1034] running on GCE node
I20260926 06:57:39.927553 7880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:39.927717 7880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:39.928843 7880 hybrid_clock.cc:648] HybridClock initialized: now 1790405859928835 us; error 26 us; skew 500 ppm
Sep 26 06:57:39 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:39.930189 7880 init.cc:377] Logged in from keytab as oryx/127.1.74.125@KRBTEST.COM (short username oryx)
I20260926 06:57:39.930465 7880 webserver.cc:460] Webserver started at http://127.1.74.125:37269/ using document root <none> and password file <none>
I20260926 06:57:39.930603 7880 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:39.930644 7880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:39.931299 7880 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:39.931864 7997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:39.931989 7880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:39.932045 7880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal
uuid: "c030d44f0c72420495cd8c27f9b508ae"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "bfa22d849ba6e4283224fec3635c9835"
server_key_iv: "5399fe429f1338817f22d3e0a9cbca98"
server_key_version: "encryptionkey@0"
I20260926 06:57:39.932104 7880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:39.940943 7880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:39.942456 7880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:39.945492 7880 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.125:40745
I20260926 06:57:39.945683 8049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.125:40745 every 8 connection(s)
I20260926 06:57:39.945884 7880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
I20260926 06:57:39.946556 8050 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:57:39.946961 8050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Bootstrap starting.
I20260926 06:57:39.949196 8050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:57:39.949427 8050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Bootstrap complete.
I20260926 06:57:39.949636 8050 raft_consensus.cc:348] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } }
I20260926 06:57:39.949702 8050 raft_consensus.cc:729] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c030d44f0c72420495cd8c27f9b508ae, State: Initialized, Role: LEARNER
I20260926 06:57:39.949729 8050 consensus_queue.cc:260] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } }
I20260926 06:57:39.949894 8053 sys_catalog.cc:455] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } } }
I20260926 06:57:39.949966 8053 sys_catalog.cc:458] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: This master's current role is: LEARNER
I20260926 06:57:39.949993 8050 sys_catalog.cc:564] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:39.951402 8064 catalog_manager.cc:1259] Loaded cluster ID: 7657cceb74fa4fb5b423da56bfabfe83
I20260926 06:57:39.951454 8064 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: loading cluster ID for follower catalog manager: success
I20260926 06:57:39.952466 8064 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: acquiring CA information for follower catalog manager: success
I20260926 06:57:39.952736 8064 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:57:39.964715 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:39.957519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.1.74.125:40745)
0926 06:57:39.957644 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:39.957647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:39.957658 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:57:39.957684 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:39.957685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:39.957698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:39.957777 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:39.958311 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.959340 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.960111 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:39.960419 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:39.960498 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:39.961832 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:39.961833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:39.961834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:39.961842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:39.963623 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.963940 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.963941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.963942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.964003 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:39.964188 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:39.964189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:39.964189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:39.964323 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:39.964423 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:39.964512 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 06:57:39.964594 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.1.74.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.124.keytab.
Entry for principal oryx/127.1.74.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.124.keytab.
I20260926 06:57:39.988204 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.124:38331
--webserver_interface=127.1.74.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.124.keytab
--principal=oryx/127.1.74.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:44973,127.1.74.125:40745,127.1.74.124:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:40.070600 8068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:40.070811 8068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:40.070833 8068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:40.072134 8068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:40.072185 8068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:40.072201 8068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:40.072213 8068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:40.072254 8068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:40.073799 8068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:44973,127.1.74.125:40745,127.1.74.124:38331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.124.keytab
--principal=oryx/127.1.74.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.124:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.74.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:40.074056 8068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:40.074302 8068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:40.079609 8073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:40.079725 8074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:40.079694 8068 server_base.cc:1034] running on GCE node
W20260926 06:57:40.079617 8076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:40.080071 8068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:40.080386 8068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:40.081528 8068 hybrid_clock.cc:648] HybridClock initialized: now 1790405860081507 us; error 40 us; skew 500 ppm
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:40.084198 8068 init.cc:377] Logged in from keytab as oryx/127.1.74.124@KRBTEST.COM (short username oryx)
I20260926 06:57:40.084661 8068 webserver.cc:460] Webserver started at http://127.1.74.124:41043/ using document root <none> and password file <none>
I20260926 06:57:40.084896 8068 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:40.084964 8068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:40.085114 8068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:40.086154 8068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data/instance:
uuid: "d8fa9a5917714233bef295f611c589ab"
format_stamp: "Formatted at 2026-09-26 06:57:40 on dist-test-slave-sfvt"
server_key: "ff16a1b299584b2c99891bff2e9b0623"
server_key_iv: "93de61384717047c794e6435a991170d"
server_key_version: "encryptionkey@0"
I20260926 06:57:40.086517 8068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal/instance:
uuid: "d8fa9a5917714233bef295f611c589ab"
format_stamp: "Formatted at 2026-09-26 06:57:40 on dist-test-slave-sfvt"
server_key: "ff16a1b299584b2c99891bff2e9b0623"
server_key_iv: "93de61384717047c794e6435a991170d"
server_key_version: "encryptionkey@0"
I20260926 06:57:40.087910 8068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:40.088650 8083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.088802 8068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:40.088904 8068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal
uuid: "d8fa9a5917714233bef295f611c589ab"
format_stamp: "Formatted at 2026-09-26 06:57:40 on dist-test-slave-sfvt"
server_key: "ff16a1b299584b2c99891bff2e9b0623"
server_key_iv: "93de61384717047c794e6435a991170d"
server_key_version: "encryptionkey@0"
I20260926 06:57:40.088977 8068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:40.104974 8068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:40.106822 8068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:40.106995 8068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:40.111178 8068 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.124:38331
I20260926 06:57:40.111212 8135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.124:38331 every 8 connection(s)
I20260926 06:57:40.111680 8068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
I20260926 06:57:40.112684 8136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:40.112841 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 8068
I20260926 06:57:40.115154 8136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.124@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.74.124@KRBTEST.COM
I20260926 06:57:40.124622 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.116015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.1.74.126:44973)
0926 06:57:40.116067 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.116069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.116099 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:57:40.116183 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.116184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.116194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.116243 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.116933 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.117487 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.118429 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.118737 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.118813 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.121200 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.121201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.121202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.121210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.123078 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.123577 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.123578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.123579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.123642 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.123929 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.123930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.123930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.124035 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.124123 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.124440 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.124517 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:57:40.124863 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.115848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34528)
0926 06:57:40.116003 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.116046 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.116127 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.116268 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.116270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.116515 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.116604 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.116608 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.117590 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.117591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.118324 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.118327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.118829 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.118830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.118908 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.119370 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.119382 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.121111 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.123169 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.123174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.123176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.123507 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.123711 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.123712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.123712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.123857 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.124123 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.124128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.124293 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.124405 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.124527 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260926 06:57:40.124990 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.113512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.1.74.124:38331)
0926 06:57:40.113686 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.113695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.113727 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:57:40.114252 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.114258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.114321 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.114444 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.115216 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.116925 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.117734 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.118135 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.118213 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.121831 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.121839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.121839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.121846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.123727 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.124087 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.124090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.124091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.124147 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.124342 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.124342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.124343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.124518 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.124673 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.124763 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.124831 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260926 06:57:40.125505 8136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.124@KRBTEST.COM for oryx/127.1.74.125@KRBTEST.COM
I20260926 06:57:40.131721 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.125801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.0.0.1:35028)
0926 06:57:40.125858 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.125870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.125913 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.126028 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.126029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.126133 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.126176 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.126177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.126797 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.126797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.127348 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.127350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.127651 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.127651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.127699 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.128116 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.128120 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.129228 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.130655 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.130656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.130656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.130823 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.131030 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.131031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.131032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.131089 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.131394 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.131395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.131435 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.131530 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.131624 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 06:57:40.131729 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.125883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.1.74.125:40745)
0926 06:57:40.125939 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.125940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.125947 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:40.125972 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.125973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.125984 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.126027 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.126230 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.126732 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.127403 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.127590 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.127641 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.129336 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.129337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.129337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.129344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.130553 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.130909 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.130910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.130910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.130967 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.131156 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.131157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.131158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.131313 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.131402 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.131558 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.131648 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:57:40.132278 8136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.124@KRBTEST.COM for oryx/127.1.74.124@KRBTEST.COM
I20260926 06:57:40.138441 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.132566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.1.74.124:38331)
0926 06:57:40.132603 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.132604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.132620 (+ 16us) server_negotiation.cc:407] Connection header received
0926 06:57:40.132677 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.132677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.132684 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.132728 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.132947 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.133522 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.134122 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.134312 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.134359 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.136039 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.136040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.136040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.136046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.137498 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.137826 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.137827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.137827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.137872 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.138016 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.138016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.138017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.138108 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.138153 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.138315 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.138377 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:57:40.138574 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.132518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.0.0.1:42786)
0926 06:57:40.132579 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.132589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.132623 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.132729 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.132730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.132849 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.132878 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.132879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.133593 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.133594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.134054 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.134056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.134370 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.134370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.134405 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.134720 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.134724 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.135964 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.137563 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.137564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.137564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.137752 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.137927 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.137927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.137928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.137975 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.138187 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.138187 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.138226 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.138297 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.138429 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":50}
I20260926 06:57:40.140280 8136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: Bootstrap starting.
I20260926 06:57:40.140942 8136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:40.141243 8136 log.cc:824] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:40.142020 8136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: No bootstrap required, opened a new log
I20260926 06:57:40.143514 8136 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } }
I20260926 06:57:40.143649 8136 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:40.143673 8136 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fa9a5917714233bef295f611c589ab, State: Initialized, Role: FOLLOWER
I20260926 06:57:40.143821 8136 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } }
I20260926 06:57:40.144109 8141 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } } }
I20260926 06:57:40.144162 8136 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:40.144188 8141 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:57:40.145841 8152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:40.145917 8152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:40.148267 8068 master_runner.cc:428] Detected that this master d8fa9a5917714233bef295f611c589ab is joining an existing cluster
I20260926 06:57:40.148317 8068 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.74.124:38331
I20260926 06:57:40.156978 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.149481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34538)
0926 06:57:40.149605 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.149615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.149688 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.149995 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.149995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.150264 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.150408 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.150410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.151391 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.151391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.152198 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.152200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.152709 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.152710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.152784 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.153320 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.153325 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.153771 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.155805 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.155806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.155806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.156045 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.156255 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.156255 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.156256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.156313 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.156611 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.156612 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.156656 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.156727 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.156830 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":20,"thread_start_us":72,"threads_started":1}
I20260926 06:57:40.157205 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.149339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.1.74.126:44973)
0926 06:57:40.149438 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.149440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.149665 (+ 225us) server_negotiation.cc:407] Connection header received
0926 06:57:40.149768 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.149769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.149780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.149908 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.150479 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.151316 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.152293 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.152609 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.152684 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.153869 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.153871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.153871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.153878 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.155720 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.156127 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.156128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.156128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.156190 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.156380 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.156381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.156382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.156515 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.156583 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.156771 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.156849 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.126@KRBTEST.COM for oryx/127.1.74.124@KRBTEST.COM
I20260926 06:57:40.176359 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 7478
I20260926 06:57:40.178965 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.149389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.1.74.125:40745)
0926 06:57:40.149449 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.149451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.149587 (+ 136us) server_negotiation.cc:407] Connection header received
0926 06:57:40.149696 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.149697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.149709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.149742 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.150510 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.151176 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.154740 (+ 3564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.155048 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.155125 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.156175 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.156176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.156177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.156184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.157876 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.158257 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.158258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.158258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.158315 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.176228 (+ 17913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.176230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.176230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.176434 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.176489 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.178697 (+ 2208us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.178813 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:57:40.179740 8158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.149339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.0.0.1:35040)
0926 06:57:40.149507 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.149517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.149615 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.149953 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.149954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.150274 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.150408 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.150410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.151314 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.151316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.154641 (+ 3325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.154643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.155127 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.155128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.155193 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.155633 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.155637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.156080 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.157958 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.157959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.157960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.158171 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.158389 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.158389 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.158390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.176112 (+ 17722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.178207 (+ 2095us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.178208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.178274 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.179055 (+ 781us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.179623 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":47,"thread_start_us":87,"threads_started":1}
I20260926 06:57:40.181165 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.149738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.0.0.1:42796)
0926 06:57:40.157184 (+ 7446us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.157195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.157595 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.157749 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.157750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.157910 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.157970 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.157971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.158807 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.158807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.159628 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.159630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.160101 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.160103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.160160 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.175996 (+ 15836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.176003 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.176394 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.179479 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.179480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.179480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.179768 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.180189 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.180190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.180190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.180244 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.180806 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.180807 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.180843 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.180894 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.181020 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7435,"thread_start_us":62,"threads_started":1}
I20260926 06:57:40.181706 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.149691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.1.74.124:38331)
0926 06:57:40.149729 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.149731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.157242 (+ 7511us) server_negotiation.cc:407] Connection header received
0926 06:57:40.157663 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.157664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.157673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.157700 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.158044 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.158721 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.159713 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.160002 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.160040 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.176498 (+ 16458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.176500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.176500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.176506 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.179391 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.180069 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.180070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.180071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.180131 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.180298 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.180299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.180299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.180717 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.180784 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.181493 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.181578 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:57:40.182479 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.159523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.0.0.1:42798)
0926 06:57:40.159570 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.159604 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.159651 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.163545 (+ 3894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.163546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.163735 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.163784 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.163786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.164723 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.164724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.165247 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.165249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.165714 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.165715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.165765 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.166269 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.166273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.167802 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.181403 (+ 13601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.181405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.181405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.181638 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.181839 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.181840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.181840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.181895 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.182159 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.182160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.182207 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.182253 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.182360 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 06:57:40.182806 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.159557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.1.74.124:38331)
0926 06:57:40.163371 (+ 3814us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.163373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.163384 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:57:40.163423 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.163424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.163436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.163515 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.163871 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.164643 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.165330 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.165636 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.165706 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.167901 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.167902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.167902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.167908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.181281 (+ 13373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.181718 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.181719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.181719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.181768 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.181972 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.181973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.181973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.182086 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.182591 (+ 505us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.182632 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.182713 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3779,"thread_start_us":108,"threads_started":1}
I20260926 06:57:40.183641 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:44673
--local_ip_for_outbound_sockets=127.1.74.65
--tserver_master_addrs=127.1.74.126:44973
--webserver_port=44925
--webserver_interface=127.1.74.65
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab
--principal=oryx/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.74.126:44973,127.1.74.125:40745
--tserver_master_addrs=127.1.74.126:44973,127.1.74.125:40745,127.1.74.124:38331 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:57:40.183676 7416 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.74.124:38331: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d8fa9a5917714233bef295f611c589ab"
member_type: NON_VOTER
last_known_addr {
host: "127.1.74.124"
port: 38331
}
attrs {
promote: true
}
}
dest_uuid: "83aa8acabfbb4e82b2cb782759ac993e"
cas_config_opid_index: 5
I20260926 06:57:40.183943 7416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } }
I20260926 06:57:40.184477 7976 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d8fa9a5917714233bef295f611c589ab (127.1.74.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.185513 7976 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.74.124:38331
I20260926 06:57:40.185588 7976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.185647 7976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:40.185864 8110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:40.186266 8068 master.cc:537] Master@127.1.74.124:38331 shutting down...
I20260926 06:57:40.186872 8110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 FOLLOWER]: Refusing update from remote peer 83aa8acabfbb4e82b2cb782759ac993e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 06:57:40.187382 7975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 06:57:40.188566 7402 consensus_peers.cc:487] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e -> Peer d8fa9a5917714233bef295f611c589ab (127.1.74.124:38331): Couldn't send request to peer d8fa9a5917714233bef295f611c589ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 06:57:40.189349 8068 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:40.189431 8068 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:40.189486 8068 tablet_replica.cc:330] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: stopping tablet replica
I20260926 06:57:40.190858 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.184960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.0.0.1:35044)
0926 06:57:40.185248 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.185259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.185299 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.186261 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.186262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.186263 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:40.186308 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.186310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.188026 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.188028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.189392 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.189396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.190525 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.190526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.190606 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.190635 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.190692 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.190748 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260926 06:57:40.191457 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.185441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.1.74.125:40745)
0926 06:57:40.186053 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.186056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.186065 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:40.186090 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.186091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.186113 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.186177 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:40.186426 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.187158 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.189631 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.190326 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.190397 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.190487 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.191208 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.191372 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587}
I20260926 06:57:40.191771 8024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 LEARNER]: Refusing update from remote peer 83aa8acabfbb4e82b2cb782759ac993e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 06:57:40.192040 7975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 06:57:40.192657 8024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d8fa9a5917714233bef295f611c589ab (127.1.74.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.193833 8165 sys_catalog.cc:455] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.194005 8165 sys_catalog.cc:458] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: This master's current role is: LEARNER
I20260926 06:57:40.194090 8053 sys_catalog.cc:455] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83aa8acabfbb4e82b2cb782759ac993e. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: NON_VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: true } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.194187 8053 sys_catalog.cc:458] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: This master's current role is: LEARNER
I20260926 06:57:40.194412 7975 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: attempting to promote NON_VOTER c030d44f0c72420495cd8c27f9b508ae to VOTER
I20260926 06:57:40.194653 7975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } }
I20260926 06:57:40.195245 8024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 LEARNER]: Refusing update from remote peer 83aa8acabfbb4e82b2cb782759ac993e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 06:57:40.195364 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.195173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.0.0.1:42814)
0926 06:57:40.195238 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.195328 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 06:57:40.195550 8162 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 06:57:40.195770 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.195657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.0.0.1:42818)
0926 06:57:40.195719 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.195742 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:38331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 06:57:40.195951 7402 consensus_peers.cc:487] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e -> Peer d8fa9a5917714233bef295f611c589ab (127.1.74.124:38331): Couldn't send request to peer d8fa9a5917714233bef295f611c589ab. Status: Network error: Client connection negotiation failed: client connection to 127.1.74.124:38331: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 06:57:40.196434 7976 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c030d44f0c72420495cd8c27f9b508ae (127.1.74.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.197293 7976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.197363 7976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:40.197413 7976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.197463 7976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:40.197706 8024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c030d44f0c72420495cd8c27f9b508ae (127.1.74.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.198618 8053 sys_catalog.cc:455] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.198691 8053 sys_catalog.cc:458] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:57:40.202611 8068 master.cc:552] Master@127.1.74.124:38331 shutdown complete.
I20260926 06:57:40.202664 8068 master_runner.cc:305] Clearing existing system tablet
I20260926 06:57:40.202795 8068 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:40.202840 8068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:40.203557 8068 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:40.204074 8167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.204219 8068 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:40.204282 8068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.204308 8068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal
uuid: "d8fa9a5917714233bef295f611c589ab"
format_stamp: "Formatted at 2026-09-26 06:57:40 on dist-test-slave-sfvt"
server_key: "ff16a1b299584b2c99891bff2e9b0623"
server_key_iv: "93de61384717047c794e6435a991170d"
server_key_version: "encryptionkey@0"
I20260926 06:57:40.205091 8068 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:40.206580 8068 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 06:57:40.206650 8068 log.cc:1196] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 06:57:40.206846 8068 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: Deleting consensus metadata
I20260926 06:57:40.206965 8068 master_runner.cc:315] Copying system tablet from 127.1.74.126:44973
I20260926 06:57:40.207700 8068 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: tablet copy: Beginning tablet copy session from remote peer at address 127.1.74.126:44973
I20260926 06:57:40.214830 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.208051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.1.74.126:44973)
0926 06:57:40.208096 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.208098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.208171 (+ 73us) server_negotiation.cc:407] Connection header received
0926 06:57:40.208253 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.208254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.208265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.208333 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.208668 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.209438 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.210496 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.210776 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.210851 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.211970 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.211972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.211972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.211980 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.213602 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.214003 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.214004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.214004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.214067 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.214272 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.214272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.214273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.214403 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.214483 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.214656 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.214734 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:57:40.214851 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.207993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34540)
0926 06:57:40.208111 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.208122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.208187 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.208338 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.208338 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.208544 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.208591 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.208593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.209534 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.209535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.210401 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.210404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.210862 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.210863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.210919 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.211471 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.211476 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.211874 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.213690 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.213691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.213691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.213926 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.214138 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.214139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.214139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.214206 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.214496 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.214497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.214546 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.214620 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.214729 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260926 06:57:40.215370 7436 tablet_copy_service.cc:140] P 83aa8acabfbb4e82b2cb782759ac993e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d8fa9a5917714233bef295f611c589ab ({username='oryx', principal='oryx/127.1.74.124@KRBTEST.COM'} at 127.0.0.1:34540)
I20260926 06:57:40.215427 7436 tablet_copy_service.cc:161] P 83aa8acabfbb4e82b2cb782759ac993e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d8fa9a5917714233bef295f611c589ab at {username='oryx', principal='oryx/127.1.74.124@KRBTEST.COM'} at 127.0.0.1:34540: session id = d8fa9a5917714233bef295f611c589ab-00000000000000000000000000000000
I20260926 06:57:40.215987 7436 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 06:57:40.216480 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 06:57:40.217805 8068 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: tablet copy: Starting download of 0 data blocks...
I20260926 06:57:40.218024 8068 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: tablet copy: Starting download of 1 WAL segments...
I20260926 06:57:40.218941 8068 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 06:57:40.220199 7436 tablet_copy_service.cc:342] P 83aa8acabfbb4e82b2cb782759ac993e: Request end of tablet copy session d8fa9a5917714233bef295f611c589ab-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.74.124@KRBTEST.COM'} at 127.0.0.1:34540
I20260926 06:57:40.220255 7436 tablet_copy_service.cc:433] P 83aa8acabfbb4e82b2cb782759ac993e: ending tablet copy session d8fa9a5917714233bef295f611c589ab-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d8fa9a5917714233bef295f611c589ab
I20260926 06:57:40.223083 8068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:40.224423 8174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:40.224694 8177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:40.224923 8068 server_base.cc:1034] running on GCE node
W20260926 06:57:40.227428 8175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:40.227620 8068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:40.227757 8068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:40.228884 8068 hybrid_clock.cc:648] HybridClock initialized: now 1790405860228861 us; error 38 us; skew 500 ppm
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:40.230656 8068 init.cc:377] Logged in from keytab as oryx/127.1.74.124@KRBTEST.COM (short username oryx)
I20260926 06:57:40.230913 8068 webserver.cc:460] Webserver started at http://127.1.74.124:40179/ using document root <none> and password file <none>
I20260926 06:57:40.231033 8068 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:40.231071 8068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:40.231945 8068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:57:40.232574 8184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.232761 8068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:40.232823 8068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal
uuid: "d8fa9a5917714233bef295f611c589ab"
format_stamp: "Formatted at 2026-09-26 06:57:40 on dist-test-slave-sfvt"
server_key: "ff16a1b299584b2c99891bff2e9b0623"
server_key_iv: "93de61384717047c794e6435a991170d"
server_key_version: "encryptionkey@0"
I20260926 06:57:40.232898 8068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:40.243086 8068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:40.245054 8068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:40.247996 8068 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.124:38331
I20260926 06:57:40.248373 8068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
I20260926 06:57:40.251456 8236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.124:38331 every 8 connection(s)
I20260926 06:57:40.252324 8237 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:57:40.252828 8237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: Bootstrap starting.
I20260926 06:57:40.255740 8237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:57:40.255981 8237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: Bootstrap complete.
I20260926 06:57:40.256182 8237 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } }
I20260926 06:57:40.256270 8237 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8fa9a5917714233bef295f611c589ab, State: Initialized, Role: LEARNER
I20260926 06:57:40.256304 8237 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } }
I20260926 06:57:40.256472 8237 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:40.257916 8251 catalog_manager.cc:1259] Loaded cluster ID: 7657cceb74fa4fb5b423da56bfabfe83
I20260926 06:57:40.258628 8251 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: loading cluster ID for follower catalog manager: success
I20260926 06:57:40.259359 8240 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.259461 8240 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: This master's current role is: LEARNER
I20260926 06:57:40.260025 8251 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: acquiring CA information for follower catalog manager: success
I20260926 06:57:40.260368 8251 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 06:57:40.287147 8163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:40.287439 8163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:40.287472 8163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:40.288712 8163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:40.288759 8163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:40.288787 8163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:40.290359 8163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab
--principal=oryx/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:44673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=44925
--tserver_master_addrs=127.1.74.126:44973,127.1.74.125:40745,127.1.74.124:38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:40.290624 8163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:40.290870 8163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:40.296936 8259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:40.297048 8163 server_base.cc:1034] running on GCE node
W20260926 06:57:40.296962 8256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:40.296969 8257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:40.297448 8163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:40.297693 8163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:40.298864 8163 hybrid_clock.cc:648] HybridClock initialized: now 1790405860298844 us; error 37 us; skew 500 ppm
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:40.301607 8163 init.cc:377] Logged in from keytab as oryx/127.1.74.65@KRBTEST.COM (short username oryx)
I20260926 06:57:40.302114 8163 webserver.cc:460] Webserver started at http://127.1.74.65:44925/ using document root <none> and password file <none>
I20260926 06:57:40.302366 8163 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:40.302443 8163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:40.303886 8163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:57:40.304476 8266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.304625 8163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:40.304721 8163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "0bcbe55244ab41129e2ebf22ccfdac68"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "544ad3dd3f1033ad29055c034fecccb7"
server_key_iv: "7552be73a4fea69100d05e71e949aaf2"
server_key_version: "encryptionkey@0"
I20260926 06:57:40.305071 8163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:40.318876 8163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:40.320832 8163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:40.320999 8163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:40.321218 8163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:40.321512 8163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:40.321543 8163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.321564 8163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:40.321609 8163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.327811 8163 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:44673
I20260926 06:57:40.327857 8379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:44673 every 8 connection(s)
I20260926 06:57:40.328184 8163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:40.330569 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 8163
I20260926 06:57:40.330737 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 7612
I20260926 06:57:40.336817 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:42713
--local_ip_for_outbound_sockets=127.1.74.66
--tserver_master_addrs=127.1.74.126:44973
--webserver_port=35155
--webserver_interface=127.1.74.66
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab
--principal=oryx/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.74.126:44973,127.1.74.125:40745
--tserver_master_addrs=127.1.74.126:44973,127.1.74.125:40745,127.1.74.124:38331 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.125@KRBTEST.COM
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.124@KRBTEST.COM
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:40.341707 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.328917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:42913 (local address 127.1.74.125:40745)
0926 06:57:40.328967 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.328969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.329260 (+ 291us) server_negotiation.cc:407] Connection header received
0926 06:57:40.330357 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.330358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.330372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.330428 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.332102 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.332856 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.334380 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.334754 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.334825 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.338376 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.338377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.338378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.338386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.340067 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.340579 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.340580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.340581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.340647 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.340933 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.340934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.340934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.341096 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.341141 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.341523 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.341604 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:57:40.341989 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.329220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:42005 (local address 127.1.74.126:44973)
0926 06:57:40.329281 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.329283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.329497 (+ 214us) server_negotiation.cc:407] Connection header received
0926 06:57:40.330406 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.330407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.330420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.330450 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.334643 (+ 4193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.335409 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.336466 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.336776 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.336819 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.339135 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.339136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.339137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.339145 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.340795 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.341184 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.341185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.341185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.341239 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.341429 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.341430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.341431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.341553 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.341596 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.341800 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.341887 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:57:40.342495 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.328862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.1.74.65:42913)
0926 06:57:40.329201 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.329211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.330227 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.330567 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.330572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.331685 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.331988 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.331994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.332977 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.332978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.334283 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.334285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.334882 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.334883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.334960 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.335983 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.335994 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.338248 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.340167 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.340174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.340176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.340484 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.340725 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.340726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.340726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.340857 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.341187 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.341193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.341371 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.341975 (+ 604us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.342257 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":64,"thread_start_us":89,"threads_started":1}
I20260926 06:57:40.342495 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.329155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.65:42005)
0926 06:57:40.329440 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.329448 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.330252 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.330567 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.330572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.334499 (+ 3927us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.334554 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.334556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.335519 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.335520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.336349 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.336352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.336900 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.336901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.336961 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.337347 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.337353 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.339032 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.340884 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.340885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.340885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.341096 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.341305 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.341306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.341306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.341359 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.341643 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.341643 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.341690 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.342072 (+ 382us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.342211 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":55,"threads_started":1}
I20260926 06:57:40.342998 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.328855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.1.74.65:33799)
0926 06:57:40.329034 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.329077 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.330227 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.330567 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.330572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.331685 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.332018 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.332020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.333282 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.333283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.334283 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.334285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.334821 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.334822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.334959 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.336092 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.336097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.339031 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.341769 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.341770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.341771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.342019 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.342302 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.342303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.342303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.342380 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.342668 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.342669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.342709 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.342779 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.342878 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":19,"thread_start_us":88,"threads_started":1}
I20260926 06:57:40.343432 8381 heartbeater.cc:347] Connected to a master server at 127.1.74.126:44973
I20260926 06:57:40.343453 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.328895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:33799 (local address 127.1.74.124:38331)
0926 06:57:40.329113 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.329116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.329358 (+ 242us) server_negotiation.cc:407] Connection header received
0926 06:57:40.330412 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.330414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.330431 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.330494 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.332107 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.333201 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.334381 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.334693 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.334750 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.339128 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.339129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.339129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.339137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.341671 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.342164 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.342165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.342165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.342228 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.342453 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.342453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.342454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.342602 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.342684 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.343217 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.343337 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":66,"threads_started":1}
I20260926 06:57:40.343531 8381 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:40.343717 8381 heartbeater.cc:511] Master 127.1.74.126:44973 requested a full tablet report, sending...
I20260926 06:57:40.343801 8380 heartbeater.cc:347] Connected to a master server at 127.1.74.124:38331
I20260926 06:57:40.343863 8380 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:40.343947 8380 heartbeater.cc:511] Master 127.1.74.124:38331 requested a full tablet report, sending...
I20260926 06:57:40.344645 8198 ts_manager.cc:194] Registered new tserver with Master: 0bcbe55244ab41129e2ebf22ccfdac68 (127.1.74.65:44673)
I20260926 06:57:40.344807 8382 heartbeater.cc:347] Connected to a master server at 127.1.74.125:40745
I20260926 06:57:40.344854 8382 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:40.344913 8382 heartbeater.cc:511] Master 127.1.74.125:40745 requested a full tablet report, sending...
I20260926 06:57:40.345208 7416 ts_manager.cc:194] Re-registered known tserver with Master: 0bcbe55244ab41129e2ebf22ccfdac68 (127.1.74.65:44673)
I20260926 06:57:40.345247 8014 ts_manager.cc:194] Registered new tserver with Master: 0bcbe55244ab41129e2ebf22ccfdac68 (127.1.74.65:44673)
I20260926 06:57:40.345755 7416 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:42005
W20260926 06:57:40.423137 8388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:40.423408 8388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:40.423451 8388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:40.424615 8388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:40.424687 8388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:40.424731 8388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:40.426214 8388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab
--principal=oryx/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:42713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=35155
--tserver_master_addrs=127.1.74.126:44973,127.1.74.125:40745,127.1.74.124:38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:40.426481 8388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:40.426707 8388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:40.431579 8394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:40.432029 8396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:40.432165 8393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:40.432425 8388 server_base.cc:1034] running on GCE node
I20260926 06:57:40.432631 8388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:40.432941 8388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:40.434129 8388 hybrid_clock.cc:648] HybridClock initialized: now 1790405860434112 us; error 32 us; skew 500 ppm
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:40.436756 8388 init.cc:377] Logged in from keytab as oryx/127.1.74.66@KRBTEST.COM (short username oryx)
I20260926 06:57:40.437199 8388 webserver.cc:460] Webserver started at http://127.1.74.66:35155/ using document root <none> and password file <none>
I20260926 06:57:40.437443 8388 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:40.437506 8388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:40.438800 8388 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:40.439388 8403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.439544 8388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:57:40.439627 8388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "c59a69a04ee3450289da29995b78644c"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "02320d4960b7801e4a768ab820ceafa6"
server_key_iv: "03086fcb121cee99822ffe15874f0fd5"
server_key_version: "encryptionkey@0"
I20260926 06:57:40.439980 8388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:40.454571 8388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:40.456668 8388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:40.456851 8388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:40.457094 8388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:40.457396 8388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:40.457458 8388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.457515 8388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:40.457540 8388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.463583 8388 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:42713
I20260926 06:57:40.463660 8516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:42713 every 8 connection(s)
I20260926 06:57:40.464015 8388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.125@KRBTEST.COM
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.124@KRBTEST.COM
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:40.472513 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 8388
I20260926 06:57:40.472666 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 7746
I20260926 06:57:40.476039 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.464735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:56543 (local address 127.1.74.124:38331)
0926 06:57:40.464789 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.464791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.465039 (+ 248us) server_negotiation.cc:407] Connection header received
0926 06:57:40.466112 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.466113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.466125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.466158 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.466858 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.467440 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.469083 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.469304 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.469391 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.472663 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.472664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.472665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.472672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.474396 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.474890 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.474891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.474892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.474953 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.475297 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.475298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.475298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.475420 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.475466 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.475856 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.475940 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:57:40.476037 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.465176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:46145 (local address 127.1.74.125:40745)
0926 06:57:40.465257 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.465259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.465276 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:57:40.466091 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.466092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.466105 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.466160 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.467341 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.467891 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.468843 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.469074 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.469133 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.471639 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.471640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.471641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.471647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.473378 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.475000 (+ 1622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.475001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.475001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.475059 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.475284 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.475285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.475286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.475420 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.475466 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.475856 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.475940 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:57:40.476619 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.464738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.1.74.66:56543)
0926 06:57:40.464901 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.464962 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.466011 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.466244 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.466244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.466611 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.466754 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.466760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.467578 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.467578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.468973 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.468975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.469392 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.469392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.469454 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.470007 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.470012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.472557 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.474502 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.474503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.474503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.474806 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.475054 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.475054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.475055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.475185 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.475518 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.475524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.475682 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.476214 (+ 532us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.476396 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":42,"thread_start_us":102,"threads_started":1}
I20260926 06:57:40.476922 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.464738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.1.74.66:46145)
0926 06:57:40.464901 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.464964 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.465985 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.466234 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.466237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.466551 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.466753 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.466757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.467974 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.467975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.468728 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.468729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.469192 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.469193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.469340 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.469846 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.469854 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.471534 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.473483 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.473491 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.473493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.474933 (+ 1440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.475129 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.475130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.475130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.475185 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.475524 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.475524 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.475682 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.476496 (+ 814us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.476578 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260926 06:57:40.477106 8517 heartbeater.cc:347] Connected to a master server at 127.1.74.124:38331
I20260926 06:57:40.477331 8517 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:40.477522 8517 heartbeater.cc:511] Master 127.1.74.124:38331 requested a full tablet report, sending...
I20260926 06:57:40.477705 8519 heartbeater.cc:347] Connected to a master server at 127.1.74.125:40745
I20260926 06:57:40.477752 8519 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:40.477818 8519 heartbeater.cc:511] Master 127.1.74.125:40745 requested a full tablet report, sending...
I20260926 06:57:40.477999 8198 ts_manager.cc:194] Registered new tserver with Master: c59a69a04ee3450289da29995b78644c (127.1.74.66:42713)
I20260926 06:57:40.478091 8014 ts_manager.cc:194] Registered new tserver with Master: c59a69a04ee3450289da29995b78644c (127.1.74.66:42713)
I20260926 06:57:40.478686 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.465275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:48693 (local address 127.1.74.126:44973)
0926 06:57:40.465318 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.465320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.465330 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:40.466111 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.466113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.466125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.466158 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.466864 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.467463 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.469063 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.469351 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.469415 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.475032 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.475033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.475034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.475042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.476814 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.477209 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.477210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.477211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.477263 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.477685 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.477686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.477686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.477833 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.477893 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.478556 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.478625 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:57:40.478788 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.465055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.66:48693)
0926 06:57:40.465165 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.465174 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.466006 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.466244 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.466244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.466615 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.466754 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.466760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.467578 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.467578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.468981 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.468983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.469469 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.469470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.469536 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.470186 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.470191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.474746 (+ 4555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.476902 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.476903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.476904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.477135 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.477482 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.477482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.477483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.477634 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.478383 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.478384 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.478441 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.478518 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.478634 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":52,"thread_start_us":55,"threads_started":1}
I20260926 06:57:40.479146 8518 heartbeater.cc:347] Connected to a master server at 127.1.74.126:44973
I20260926 06:57:40.479231 8518 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:40.479336 8518 heartbeater.cc:511] Master 127.1.74.126:44973 requested a full tablet report, sending...
I20260926 06:57:40.479609 7416 ts_manager.cc:194] Re-registered known tserver with Master: c59a69a04ee3450289da29995b78644c (127.1.74.66:42713)
I20260926 06:57:40.479640 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:39347
--local_ip_for_outbound_sockets=127.1.74.67
--tserver_master_addrs=127.1.74.126:44973
--webserver_port=42791
--webserver_interface=127.1.74.67
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab
--principal=oryx/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.74.126:44973,127.1.74.125:40745
--tserver_master_addrs=127.1.74.126:44973,127.1.74.125:40745,127.1.74.124:38331 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:57:40.480096 7416 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:48693
W20260926 06:57:40.563628 8524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:40.563870 8524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:40.563915 8524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:40.565016 8524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:40.565049 8524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:40.565076 8524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:40.566483 8524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab
--principal=oryx/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:39347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=42791
--tserver_master_addrs=127.1.74.126:44973,127.1.74.125:40745,127.1.74.124:38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:40.566665 8524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:40.566838 8524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:40.571798 8532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:40.571861 8530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:40.572042 8524 server_base.cc:1034] running on GCE node
W20260926 06:57:40.571882 8529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:40.572263 8524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:40.572468 8524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:40.573614 8524 hybrid_clock.cc:648] HybridClock initialized: now 1790405860573584 us; error 44 us; skew 500 ppm
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:40.576347 8524 init.cc:377] Logged in from keytab as oryx/127.1.74.67@KRBTEST.COM (short username oryx)
I20260926 06:57:40.576788 8524 webserver.cc:460] Webserver started at http://127.1.74.67:42791/ using document root <none> and password file <none>
I20260926 06:57:40.577009 8524 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:40.577062 8524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:40.578402 8524 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:40.578958 8539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.579133 8524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:40.579219 8524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "376e271855ce4c63bc91e29cae1041da"
format_stamp: "Formatted at 2026-09-26 06:57:39 on dist-test-slave-sfvt"
server_key: "d23fa5a52b780d917841282cd9cc24fa"
server_key_iv: "bf96dd86040867400559528ed1487067"
server_key_version: "encryptionkey@0"
I20260926 06:57:40.579614 8524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:40.590994 8524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:40.592839 8524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:40.592948 8524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:40.593122 8524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:40.593434 8524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:40.593465 8524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.593487 8524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:40.593500 8524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:40.599372 8524 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:39347
I20260926 06:57:40.599468 8652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:39347 every 8 connection(s)
I20260926 06:57:40.599785 8524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:57:40.604523 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 8524
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.125@KRBTEST.COM
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.124@KRBTEST.COM
Sep 26 06:57:40 dist-test-slave-sfvt krb5kdc[7371](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:40.613982 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.608978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.0.0.1:42834)
0926 06:57:40.609739 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.609749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.609786 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.610658 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.610659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.610661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:40.610704 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.610706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.611669 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.611670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.612805 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.612807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.613729 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.613730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.613801 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.613805 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.613834 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.613881 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743}
I20260926 06:57:40.614104 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.600547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:35003 (local address 127.1.74.125:40745)
0926 06:57:40.600594 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.600596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.600803 (+ 207us) server_negotiation.cc:407] Connection header received
0926 06:57:40.601921 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.601922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.601934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.601968 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.602794 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.603573 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.604710 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.605006 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.605049 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.609353 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.609354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.609354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.609361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.611202 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.612794 (+ 1592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.612795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.612796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.612864 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.613305 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.613306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.613307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.613450 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.613509 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.613915 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.613989 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:57:40.614665 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.600496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.1.74.67:35003)
0926 06:57:40.600749 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.600757 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.601796 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.602065 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.602070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.602521 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.602686 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.602691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.603665 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.603665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.604610 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.604612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.605129 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.605130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.605198 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.605834 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.605838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.609245 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.611675 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.611683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.611686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.612735 (+ 1049us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.613104 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.613105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.613105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.613242 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.613570 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.613576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.613750 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.614194 (+ 444us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.614389 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":61,"thread_start_us":87,"threads_started":1}
I20260926 06:57:40.615168 8655 heartbeater.cc:347] Connected to a master server at 127.1.74.125:40745
I20260926 06:57:40.615295 8655 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:40.615474 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.609880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.1.74.124:38331)
0926 06:57:40.610498 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.610499 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.610506 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:40.610526 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.610527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.610557 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.610765 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:40.610794 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.611567 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.612943 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.613633 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.614583 (+ 950us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.614669 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.614690 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.615189 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":56,"threads_started":1}
I20260926 06:57:40.615518 8655 heartbeater.cc:511] Master 127.1.74.125:40745 requested a full tablet report, sending...
I20260926 06:57:40.615996 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.600478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:59039 (local address 127.1.74.124:38331)
0926 06:57:40.600540 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.600542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.600768 (+ 226us) server_negotiation.cc:407] Connection header received
0926 06:57:40.601921 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.601922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.601934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.601968 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.602793 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.603578 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.604710 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.605016 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.605052 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.610296 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.610297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.610298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.610317 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.612545 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.613343 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.613344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.613344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.613391 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.614939 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.614940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.614940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.615037 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.615069 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.615827 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.615897 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:57:40.615939 8211 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 06:57:40.616011 8014 ts_manager.cc:194] Registered new tserver with Master: 376e271855ce4c63bc91e29cae1041da (127.1.74.67:39347)
I20260926 06:57:40.616117 8240 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83aa8acabfbb4e82b2cb782759ac993e. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.616199 8240 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: This master's current role is: LEARNER
I20260926 06:57:40.616402 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.605718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.1.74.126:44973)
0926 06:57:40.606912 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.606914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.606921 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:40.606940 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.606941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.606951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.607783 (+ 832us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.607824 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.608649 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.609995 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.610304 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.610407 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.612870 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.612871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.612872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.612877 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.614873 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.615317 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.615317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.615318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.615373 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.615592 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.615593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.615594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.615710 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.616169 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.616212 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.616303 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1169,"thread_start_us":86,"threads_started":1}
I20260926 06:57:40.616475 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.600497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.1.74.67:59039)
0926 06:57:40.600649 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.600689 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.601796 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.602066 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.602070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.602500 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.602686 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.602691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.603665 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.603665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.604611 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.604612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.605130 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.605131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.605198 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.605767 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.605780 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.610179 (+ 4399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.612831 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.612832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.612832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.613036 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.613462 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.613463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.613463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.613515 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.615121 (+ 1606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.615121 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.615706 (+ 585us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.615959 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.616348 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":49,"thread_start_us":97,"threads_started":1}
I20260926 06:57:40.615604 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.605918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.1.74.124:38331)
0926 06:57:40.606584 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.606586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.606744 (+ 158us) server_negotiation.cc:407] Connection header received
0926 06:57:40.606767 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.606768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.606780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.606840 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.607682 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.608548 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.609673 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.610033 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.610081 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.612491 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.612492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.612492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.612499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.614196 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.614579 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.614580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.614580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.614632 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.614861 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.614862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.614863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.615000 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.615075 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.615448 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.615522 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":54,"threads_started":1}
I20260926 06:57:40.616775 8653 heartbeater.cc:347] Connected to a master server at 127.1.74.124:38331
I20260926 06:57:40.616811 8653 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:40.616871 8653 heartbeater.cc:511] Master 127.1.74.124:38331 requested a full tablet report, sending...
I20260926 06:57:40.617159 8198 ts_manager.cc:194] Registered new tserver with Master: 376e271855ce4c63bc91e29cae1041da (127.1.74.67:39347)
I20260926 06:57:40.617892 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.600855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:55971 (local address 127.1.74.126:44973)
0926 06:57:40.600900 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.600903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.600968 (+ 65us) server_negotiation.cc:407] Connection header received
0926 06:57:40.601904 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.601905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.601917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.601978 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.602781 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.603329 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.604360 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.604659 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.604707 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.609237 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.609238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.609239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.609246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.612235 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.612680 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.612681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.612681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.612983 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.617371 (+ 4388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.617372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.617373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.617493 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.617545 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.617746 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.617824 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:57:40.617920 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.600784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.67:55971)
0926 06:57:40.600913 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.600923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.601807 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.602067 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.602070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.602483 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:40.602685 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.602689 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.603432 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.603433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.604230 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.604232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.604811 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.604813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.604959 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.605909 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:40.605914 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:40.609131 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:40.612333 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.612334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.612335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.612602 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.616955 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:40.616956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:40.616956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:40.617024 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:40.617600 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:40.617601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:40.617640 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.617709 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.617808 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260926 06:57:40.618378 8654 heartbeater.cc:347] Connected to a master server at 127.1.74.126:44973
I20260926 06:57:40.618430 8654 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:40.618486 8654 heartbeater.cc:511] Master 127.1.74.126:44973 requested a full tablet report, sending...
I20260926 06:57:40.618716 7415 ts_manager.cc:194] Re-registered known tserver with Master: 376e271855ce4c63bc91e29cae1041da (127.1.74.67:39347)
I20260926 06:57:40.619153 7415 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:55971
I20260926 06:57:40.618387 7416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:40.620710 7416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:40.621639 8162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: NON_VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: true } } }
I20260926 06:57:40.621733 8162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:40.623242 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.606443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.1.74.125:40745)
0926 06:57:40.606551 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.606553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.607288 (+ 735us) server_negotiation.cc:407] Connection header received
0926 06:57:40.607360 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.607361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.607373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.607431 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.608134 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.617208 (+ 9074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.618292 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.618623 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.618680 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.620286 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.620287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.620288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.620293 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.622015 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.622365 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.622366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.622367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.622427 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.622654 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.622655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.622656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.622797 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.622907 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.623059 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.623146 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260926 06:57:40.623375 8162 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: attempting to promote NON_VOTER d8fa9a5917714233bef295f611c589ab to VOTER
I20260926 06:57:40.623518 8162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } }
I20260926 06:57:40.623872 8024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Refusing update from remote peer 83aa8acabfbb4e82b2cb782759ac993e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 06:57:40.624171 7975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 06:57:40.624600 8210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 LEARNER]: Refusing update from remote peer 83aa8acabfbb4e82b2cb782759ac993e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 06:57:40.624792 7975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 06:57:40.624917 7975 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d8fa9a5917714233bef295f611c589ab (127.1.74.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:40.625038 8024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d8fa9a5917714233bef295f611c589ab (127.1.74.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:40.625672 8210 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, d8fa9a5917714233bef295f611c589ab (127.1.74.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:40.625852 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:40.625941 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:40.625898 8053 sys_catalog.cc:455] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:40.625970 8053 sys_catalog.cc:458] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:57:40.625996 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:40.626060 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:40.626096 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:40.626145 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:40.626724 8240 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "83aa8acabfbb4e82b2cb782759ac993e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:40.626823 8240 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:57:40.630568 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.626966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44673 (local address 127.0.0.1:42362)
0926 06:57:40.627002 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.627013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.627051 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.627389 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.627390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.627392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:40.627436 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.627438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.628635 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.628636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.629469 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.629471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.630357 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.630358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.630422 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.630426 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.630453 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.630488 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 06:57:40.630950 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.627059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.1.74.65:44673)
0926 06:57:40.627207 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.627212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.627222 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:40.627286 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.627288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.627305 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.627393 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:40.627500 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.628564 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.629561 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.630263 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.630739 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.630809 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.630835 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.630873 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260926 06:57:40.631042 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.627078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:39347 (local address 127.0.0.1:44642)
0926 06:57:40.627382 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.627393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.627447 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.627699 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.627700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.627702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:40.627747 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.627749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.629160 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.629161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.629982 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.629984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.630760 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.630761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.630851 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.630855 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.630936 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.630976 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":81,"threads_started":1}
I20260926 06:57:40.631085 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.627154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.1.74.67:39347)
0926 06:57:40.627317 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.627321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.627504 (+ 183us) server_negotiation.cc:407] Connection header received
0926 06:57:40.627548 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.627550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.627566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.627636 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:40.628289 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.629077 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.630065 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.630584 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.630651 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.630729 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.630954 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.631007 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260926 06:57:40.631652 8314 tablet_service.cc:1467] Processing CreateTablet for tablet 86cd7596e2c4423d95ae341850416241 (DEFAULT_TABLE table=test-table [id=d4f0e3221485419aad9866504834aff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:40.631826 8587 tablet_service.cc:1467] Processing CreateTablet for tablet 86cd7596e2c4423d95ae341850416241 (DEFAULT_TABLE table=test-table [id=d4f0e3221485419aad9866504834aff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:40.632040 8314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86cd7596e2c4423d95ae341850416241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:40.632174 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86cd7596e2c4423d95ae341850416241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:40.632665 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.627242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:42713 (local address 127.0.0.1:33970)
0926 06:57:40.627668 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.627678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.627730 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.627900 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.627900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.627901 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:40.627924 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.627925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.629811 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.629812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.631421 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.631424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.632430 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.632431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.632496 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.632501 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.632560 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.632597 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":101,"threads_started":1}
I20260926 06:57:40.632882 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.627471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.1.74.66:42713)
0926 06:57:40.627650 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.627654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.627721 (+ 67us) server_negotiation.cc:407] Connection header received
0926 06:57:40.627805 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.627807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.627822 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.627899 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:40.627956 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.629716 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.631517 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.632243 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.632333 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.632393 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.632612 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.632654 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260926 06:57:40.633567 8451 tablet_service.cc:1467] Processing CreateTablet for tablet 86cd7596e2c4423d95ae341850416241 (DEFAULT_TABLE table=test-table [id=d4f0e3221485419aad9866504834aff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:40.633893 8451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86cd7596e2c4423d95ae341850416241. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:40.635321 8683 tablet_bootstrap.cc:492] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: Bootstrap starting.
I20260926 06:57:40.636021 8684 tablet_bootstrap.cc:492] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: Bootstrap starting.
I20260926 06:57:40.636631 8684 tablet_bootstrap.cc:654] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:40.635921 8683 tablet_bootstrap.cc:654] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:40.636971 8684 log.cc:824] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:40.637511 8683 log.cc:824] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:40.637895 8684 tablet_bootstrap.cc:492] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: No bootstrap required, opened a new log
I20260926 06:57:40.637948 8684 ts_tablet_manager.cc:1397] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:40.639216 8684 raft_consensus.cc:348] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:40.639438 8684 raft_consensus.cc:374] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:40.639405 8686 tablet_bootstrap.cc:492] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: Bootstrap starting.
I20260926 06:57:40.639487 8684 raft_consensus.cc:729] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e271855ce4c63bc91e29cae1041da, State: Initialized, Role: FOLLOWER
I20260926 06:57:40.639604 8684 consensus_queue.cc:260] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:40.639899 8684 ts_tablet_manager.cc:1428] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:40.639916 8654 heartbeater.cc:503] Master 127.1.74.126:44973 was elected leader, sending a full tablet report...
I20260926 06:57:40.640033 8686 tablet_bootstrap.cc:654] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:40.640322 8686 log.cc:824] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:40.641114 8683 tablet_bootstrap.cc:492] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: No bootstrap required, opened a new log
I20260926 06:57:40.641181 8683 ts_tablet_manager.cc:1397] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 06:57:40.642989 8683 raft_consensus.cc:348] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:40.643105 8683 raft_consensus.cc:374] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:40.643126 8683 raft_consensus.cc:729] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcbe55244ab41129e2ebf22ccfdac68, State: Initialized, Role: FOLLOWER
I20260926 06:57:40.643234 8683 consensus_queue.cc:260] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:40.643527 8683 ts_tablet_manager.cc:1428] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:57:40.643597 8381 heartbeater.cc:503] Master 127.1.74.126:44973 was elected leader, sending a full tablet report...
I20260926 06:57:40.643821 8686 tablet_bootstrap.cc:492] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: No bootstrap required, opened a new log
I20260926 06:57:40.643891 8686 ts_tablet_manager.cc:1397] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 06:57:40.645409 8686 raft_consensus.cc:348] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:40.645529 8686 raft_consensus.cc:374] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:40.645550 8686 raft_consensus.cc:729] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59a69a04ee3450289da29995b78644c, State: Initialized, Role: FOLLOWER
I20260926 06:57:40.645677 8686 consensus_queue.cc:260] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:40.645946 8686 ts_tablet_manager.cc:1428] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:40.645985 8518 heartbeater.cc:503] Master 127.1.74.126:44973 was elected leader, sending a full tablet report...
I20260926 06:57:40.710749 8690 raft_consensus.cc:482] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:40.710871 8690 raft_consensus.cc:504] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:40.711467 8690 leader_election.cc:288] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376e271855ce4c63bc91e29cae1041da (127.1.74.67:39347), c59a69a04ee3450289da29995b78644c (127.1.74.66:42713)
I20260926 06:57:40.714298 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.711741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:35789 (local address 127.1.74.67:39347)
0926 06:57:40.711808 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.711810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.711821 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:57:40.711913 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.711915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.711925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.711974 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:40.712139 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.712630 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.713578 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.713943 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.714040 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.714050 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.714193 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.714230 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:57:40.714298 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.711653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:39347 (local address 127.1.74.65:35789)
0926 06:57:40.711715 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.711730 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.711814 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.711977 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.711978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.711979 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:40.712059 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.712061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.712726 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.712727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.713474 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.713476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.714044 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.714045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.714117 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.714119 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.714188 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.714227 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":27}
I20260926 06:57:40.714385 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.711884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:41359 (local address 127.1.74.66:42713)
0926 06:57:40.711952 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.711954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.711962 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:40.711989 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.711990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.712001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.712076 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:40.712258 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.712744 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.713656 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.714011 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.714095 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.714104 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.714259 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.714295 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:57:40.714570 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.711795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:42713 (local address 127.1.74.65:41359)
0926 06:57:40.711841 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:40.711862 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:40.711923 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:40.712080 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:40.712081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:40.712082 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:40.712166 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.712168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.712833 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.712834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.713557 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:40.713559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.714095 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.714096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:40.714173 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.714176 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:40.714250 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:57:40.714289 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260926 06:57:40.714876 8607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86cd7596e2c4423d95ae341850416241" candidate_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e271855ce4c63bc91e29cae1041da" is_pre_election: true
I20260926 06:57:40.715062 8607 raft_consensus.cc:2454] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bcbe55244ab41129e2ebf22ccfdac68 in term 0.
I20260926 06:57:40.715205 8471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86cd7596e2c4423d95ae341850416241" candidate_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59a69a04ee3450289da29995b78644c" is_pre_election: true
I20260926 06:57:40.715340 8471 raft_consensus.cc:2454] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bcbe55244ab41129e2ebf22ccfdac68 in term 0.
W20260926 06:57:40.715556 8520 tablet.cc:2369] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:40.715564 8268 leader_election.cc:302] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcbe55244ab41129e2ebf22ccfdac68, 376e271855ce4c63bc91e29cae1041da; no voters:
I20260926 06:57:40.715734 8690 raft_consensus.cc:2781] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:40.715812 8690 raft_consensus.cc:482] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:40.715847 8690 raft_consensus.cc:3037] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:40.716593 8690 raft_consensus.cc:504] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:40.716897 8690 leader_election.cc:288] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [CANDIDATE]: Term 1 election: Requested vote from peers 376e271855ce4c63bc91e29cae1041da (127.1.74.67:39347), c59a69a04ee3450289da29995b78644c (127.1.74.66:42713)
I20260926 06:57:40.716885 8607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86cd7596e2c4423d95ae341850416241" candidate_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e271855ce4c63bc91e29cae1041da"
I20260926 06:57:40.716955 8471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86cd7596e2c4423d95ae341850416241" candidate_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59a69a04ee3450289da29995b78644c"
I20260926 06:57:40.716959 8607 raft_consensus.cc:3037] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:40.717037 8471 raft_consensus.cc:3037] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:40.717721 8607 raft_consensus.cc:2454] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bcbe55244ab41129e2ebf22ccfdac68 in term 1.
I20260926 06:57:40.717777 8471 raft_consensus.cc:2454] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bcbe55244ab41129e2ebf22ccfdac68 in term 1.
I20260926 06:57:40.717897 8268 leader_election.cc:302] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcbe55244ab41129e2ebf22ccfdac68, 376e271855ce4c63bc91e29cae1041da; no voters:
I20260926 06:57:40.718000 8690 raft_consensus.cc:2781] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:40.718173 8690 raft_consensus.cc:686] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 LEADER]: Becoming Leader. State: Replica: 0bcbe55244ab41129e2ebf22ccfdac68, State: Running, Role: LEADER
I20260926 06:57:40.718452 8690 consensus_queue.cc:237] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:40.719513 7415 catalog_manager.cc:5495] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bcbe55244ab41129e2ebf22ccfdac68 (127.1.74.65). New cstate: current_term: 1 leader_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } health_report { overall_health: UNKNOWN } } }
I20260926 06:57:40.744524 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.742304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.1.74.65:44673)
0926 06:57:40.742349 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.742351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.742359 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:40.742419 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.742419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.742438 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.742499 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:40.742602 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.743096 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.743875 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.744059 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.744113 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.744231 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:40.744327 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:40.744383 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.744442 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.744460 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:57:40.746294 8471 raft_consensus.cc:1261] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [term 1 FOLLOWER]: Refusing update from remote peer 0bcbe55244ab41129e2ebf22ccfdac68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:40.746322 8607 raft_consensus.cc:1261] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [term 1 FOLLOWER]: Refusing update from remote peer 0bcbe55244ab41129e2ebf22ccfdac68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:40.746671 8690 consensus_queue.cc:1035] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:40.746857 8692 consensus_queue.cc:1035] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:40.750031 8696 mvcc.cc:205] Tried to move back new op lower bound from 7333502405613883392 to 7333502405503168512. Current Snapshot: MvccSnapshot[applied={T|T < 7333502405613883392}]
I20260926 06:57:40.750588 8697 mvcc.cc:205] Tried to move back new op lower bound from 7333502405613883392 to 7333502405503168512. Current Snapshot: MvccSnapshot[applied={T|T < 7333502405613883392}]
I20260926 06:57:40.753013 7416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:40.753119 7416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:40.754482 7416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83aa8acabfbb4e82b2cb782759ac993e: Sending DeleteTablet for 3 replicas of tablet 86cd7596e2c4423d95ae341850416241
I20260926 06:57:40.754772 8314 tablet_service.cc:1514] Processing DeleteTablet for tablet 86cd7596e2c4423d95ae341850416241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:40 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:42362
I20260926 06:57:40.754792 8587 tablet_service.cc:1514] Processing DeleteTablet for tablet 86cd7596e2c4423d95ae341850416241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:40 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:44642
I20260926 06:57:40.754828 8451 tablet_service.cc:1514] Processing DeleteTablet for tablet 86cd7596e2c4423d95ae341850416241 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:40 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:33970
I20260926 06:57:40.754997 8706 tablet_replica.cc:330] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: stopping tablet replica
I20260926 06:57:40.755085 8706 raft_consensus.cc:2229] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:40.755177 8706 raft_consensus.cc:2258] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:40.756315 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 7384
I20260926 06:57:40.763520 8706 ts_tablet_manager.cc:1905] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:40.765410 8706 ts_tablet_manager.cc:1918] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:40.765483 8706 log.cc:1196] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/86cd7596e2c4423d95ae341850416241
I20260926 06:57:40.765672 8706 ts_tablet_manager.cc:1939] T 86cd7596e2c4423d95ae341850416241 P 0bcbe55244ab41129e2ebf22ccfdac68: Deleting consensus metadata
I20260926 06:57:40.765816 8705 tablet_replica.cc:330] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: stopping tablet replica
I20260926 06:57:40.765950 8705 raft_consensus.cc:2229] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:40.766070 8705 raft_consensus.cc:2258] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 06:57:40.766358 8269 connection.cc:441] server connection from 127.0.0.1:42362 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42362 (request call id 1) could send its response
I20260926 06:57:40.766525 8705 ts_tablet_manager.cc:1905] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:40.767414 8707 tablet_replica.cc:330] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: stopping tablet replica
I20260926 06:57:40.767526 8707 raft_consensus.cc:2229] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:40.767639 8707 raft_consensus.cc:2258] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:40.767983 8705 ts_tablet_manager.cc:1918] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:40.768055 8705 log.cc:1196] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/86cd7596e2c4423d95ae341850416241
I20260926 06:57:40.768338 8705 ts_tablet_manager.cc:1939] T 86cd7596e2c4423d95ae341850416241 P 376e271855ce4c63bc91e29cae1041da: Deleting consensus metadata
W20260926 06:57:40.768703 8542 connection.cc:441] server connection from 127.0.0.1:44642 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44642 (request call id 1) could send its response
I20260926 06:57:40.769546 8670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.755851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.1.74.124:38331)
0926 06:57:40.755896 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.755899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.756122 (+ 223us) server_negotiation.cc:407] Connection header received
0926 06:57:40.756281 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.756282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.756297 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.756341 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.757788 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.758599 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.759817 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.760113 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.760200 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.765379 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.765381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.765382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.765391 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.767572 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.768306 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.768307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.768307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.768374 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.768645 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.768646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.768646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.768793 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.768893 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.769361 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.769447 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:57:40.770258 8707 ts_tablet_manager.cc:1905] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:40.772564 8707 ts_tablet_manager.cc:1918] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:40.772703 8707 log.cc:1196] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/86cd7596e2c4423d95ae341850416241
I20260926 06:57:40.773053 8707 ts_tablet_manager.cc:1939] T 86cd7596e2c4423d95ae341850416241 P c59a69a04ee3450289da29995b78644c: Deleting consensus metadata
W20260926 06:57:40.774242 8405 connection.cc:441] server connection from 127.0.0.1:33970 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33970 (request call id 1) could send its response
I20260926 06:57:40.776134 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.764057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.1.74.125:40745)
0926 06:57:40.764116 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.764119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.768404 (+ 4285us) server_negotiation.cc:407] Connection header received
0926 06:57:40.768512 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.768513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.768529 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.768559 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.769288 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.770089 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.771082 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.771434 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.771474 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.773086 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.773088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.773089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.773097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.774731 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.775132 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.775133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.775133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.775184 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.775474 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.775475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.775476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.775621 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.775712 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.775935 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.776031 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:57:40.776372 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:40.768093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.1.74.124:38331)
0926 06:57:40.768135 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:40.768137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:40.768976 (+ 839us) server_negotiation.cc:407] Connection header received
0926 06:57:40.769074 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:40.769075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:40.769088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:40.769118 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:40.769888 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.770599 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.771644 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:40.771971 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:40.772014 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:40.773412 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:40.773413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:40.773413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:40.773420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:40.775058 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.775535 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.775536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.775536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.775583 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:40.775808 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:40.775808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:40.775808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:40.775939 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:40.776026 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:40.776196 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 06:57:40.776273 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:57:41.751223 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:41.750883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.65:57049)
0926 06:57:41.751069 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:41.751162 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":63,"threads_started":1}
W20260926 06:57:41.751701 8381 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:44973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
I20260926 06:57:41.751758 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:41.751469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.67:57639)
0926 06:57:41.751647 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:41.751715 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":48,"threads_started":1}
W20260926 06:57:41.751983 8654 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:44973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
I20260926 06:57:41.752384 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:41.752173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.66:46745)
0926 06:57:41.752277 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:41.752339 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
W20260926 06:57:41.752609 8518 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:44973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
I20260926 06:57:42.255898 8719 raft_consensus.cc:482] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83aa8acabfbb4e82b2cb782759ac993e)
I20260926 06:57:42.256050 8719 raft_consensus.cc:504] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } }
I20260926 06:57:42.256318 8719 leader_election.cc:288] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83aa8acabfbb4e82b2cb782759ac993e (127.1.74.126:44973), d8fa9a5917714233bef295f611c589ab (127.1.74.124:38331)
I20260926 06:57:42.256598 8720 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83aa8acabfbb4e82b2cb782759ac993e)
I20260926 06:57:42.256732 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.256494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34972)
0926 06:57:42.256624 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.256689 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":47,"threads_started":1}
I20260926 06:57:42.256727 8720 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } }
W20260926 06:57:42.256878 8000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
I20260926 06:57:42.257335 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.257241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34988)
0926 06:57:42.257283 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.257309 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 06:57:42.257503 8720 leader_election.cc:288] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83aa8acabfbb4e82b2cb782759ac993e (127.1.74.126:44973), c030d44f0c72420495cd8c27f9b508ae (127.1.74.125:40745)
W20260926 06:57:42.257584 8000 leader_election.cc:334] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83aa8acabfbb4e82b2cb782759ac993e (127.1.74.126:44973): Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
I20260926 06:57:42.257701 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.257079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34984)
0926 06:57:42.257622 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.257672 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":39,"threads_started":1}
W20260926 06:57:42.257827 8187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
I20260926 06:57:42.258404 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.258296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:34992)
0926 06:57:42.258352 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.258378 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 06:57:42.258498 8187 leader_election.cc:334] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83aa8acabfbb4e82b2cb782759ac993e (127.1.74.126:44973): Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
I20260926 06:57:42.259905 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.256669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:38331 (local address 127.0.0.1:42860)
0926 06:57:42.256798 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.256809 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:42.256866 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:42.257104 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:42.257105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:42.257108 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:42.257175 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.257177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.258121 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.258122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.259007 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.259009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.259643 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.259644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.259709 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.259713 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:42.259797 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 06:57:42.259839 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260926 06:57:42.259902 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.256882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.1.74.124:38331)
0926 06:57:42.256965 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:42.256968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:42.256975 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:42.257002 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:42.257003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:42.257014 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:42.257244 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:42.257282 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.258059 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.259110 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.259558 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.259643 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.259655 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:42.259804 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 06:57:42.259843 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":40,"threads_started":1}
I20260926 06:57:42.260821 8210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c030d44f0c72420495cd8c27f9b508ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d8fa9a5917714233bef295f611c589ab" is_pre_election: true
I20260926 06:57:42.260911 8210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c030d44f0c72420495cd8c27f9b508ae in term 1.
I20260926 06:57:42.261086 8000 leader_election.cc:302] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c030d44f0c72420495cd8c27f9b508ae, d8fa9a5917714233bef295f611c589ab; no voters: 83aa8acabfbb4e82b2cb782759ac993e
I20260926 06:57:42.261209 8719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 06:57:42.261248 8719 raft_consensus.cc:482] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83aa8acabfbb4e82b2cb782759ac993e)
I20260926 06:57:42.261279 8719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:57:42.261790 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.257402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.1.74.125:40745)
0926 06:57:42.257704 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:42.257706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:42.257816 (+ 110us) server_negotiation.cc:407] Connection header received
0926 06:57:42.257914 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:42.257915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:42.257925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:42.258000 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:42.259097 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.259573 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.260629 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.261448 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.261527 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.261539 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:42.261693 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:57:42.261731 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":78,"threads_started":1}
I20260926 06:57:42.261799 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.257289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:40745 (local address 127.0.0.1:35062)
0926 06:57:42.257761 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.257769 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:42.257826 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:42.258018 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:42.258019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:42.258021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:42.258056 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.258058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.259654 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.259655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.260536 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.260539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.261539 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.261540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.261604 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.261612 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 06:57:42.261687 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:57:42.261728 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":34,"threads_started":1}
I20260926 06:57:42.262034 8024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8fa9a5917714233bef295f611c589ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c030d44f0c72420495cd8c27f9b508ae" is_pre_election: true
I20260926 06:57:42.262141 8719 raft_consensus.cc:504] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } }
I20260926 06:57:42.262218 8024 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8fa9a5917714233bef295f611c589ab in current term 2: Already voted for candidate c030d44f0c72420495cd8c27f9b508ae in this term.
I20260926 06:57:42.262254 8719 leader_election.cc:288] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [CANDIDATE]: Term 2 election: Requested vote from peers 83aa8acabfbb4e82b2cb782759ac993e (127.1.74.126:44973), d8fa9a5917714233bef295f611c589ab (127.1.74.124:38331)
I20260926 06:57:42.262424 8185 leader_election.cc:302] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8fa9a5917714233bef295f611c589ab; no voters: 83aa8acabfbb4e82b2cb782759ac993e, c030d44f0c72420495cd8c27f9b508ae
I20260926 06:57:42.262475 8210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c030d44f0c72420495cd8c27f9b508ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d8fa9a5917714233bef295f611c589ab"
I20260926 06:57:42.262526 8720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:57:42.262615 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.262324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:35012)
0926 06:57:42.262565 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.262593 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20260926 06:57:42.262888 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.262792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:35024)
0926 06:57:42.262839 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.262867 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 06:57:42.262974 8000 leader_election.cc:334] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83aa8acabfbb4e82b2cb782759ac993e (127.1.74.126:44973): Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
I20260926 06:57:42.263412 8720 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:57:42.264065 8210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c030d44f0c72420495cd8c27f9b508ae in term 2.
I20260926 06:57:42.264503 8000 leader_election.cc:302] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c030d44f0c72420495cd8c27f9b508ae, d8fa9a5917714233bef295f611c589ab; no voters: 83aa8acabfbb4e82b2cb782759ac993e
I20260926 06:57:42.264598 8719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:57:42.264757 8719 raft_consensus.cc:686] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [term 2 LEADER]: Becoming Leader. State: Replica: c030d44f0c72420495cd8c27f9b508ae, State: Running, Role: LEADER
I20260926 06:57:42.264814 8719 consensus_queue.cc:237] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } }
I20260926 06:57:42.265167 8729 sys_catalog.cc:455] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c030d44f0c72420495cd8c27f9b508ae. Latest consensus state: current_term: 2 leader_uuid: "c030d44f0c72420495cd8c27f9b508ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:42.265269 8729 sys_catalog.cc:458] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:42.265461 8731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:42.265880 8731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:42.265939 8731 catalog_manager.cc:1259] Loaded cluster ID: 7657cceb74fa4fb5b423da56bfabfe83
I20260926 06:57:42.265956 8731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:42.266139 8731 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:42.266237 8731 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Loaded TSK: 0
I20260926 06:57:42.266382 8731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:57:42.317140 8012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:42.318761 8012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:42.319805 8210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [term 2 FOLLOWER]: Refusing update from remote peer c030d44f0c72420495cd8c27f9b508ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 06:57:42.319846 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.319743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:35038)
0926 06:57:42.319782 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.319809 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 06:57:42.320042 8729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 06:57:42.320202 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.320095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.0.0.1:35040)
0926 06:57:42.320129 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.320156 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 06:57:42.320254 8000 consensus_peers.cc:487] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae -> Peer 83aa8acabfbb4e82b2cb782759ac993e (127.1.74.126:44973): Couldn't send request to peer 83aa8acabfbb4e82b2cb782759ac993e. Status: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 06:57:42.320653 8720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c030d44f0c72420495cd8c27f9b508ae. Latest consensus state: current_term: 2 leader_uuid: "c030d44f0c72420495cd8c27f9b508ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:42.320756 8720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:57:42.321231 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c030d44f0c72420495cd8c27f9b508ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:42.321326 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:42.321341 8720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c030d44f0c72420495cd8c27f9b508ae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "83aa8acabfbb4e82b2cb782759ac993e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44973 } } peers { permanent_uuid: "c030d44f0c72420495cd8c27f9b508ae" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 40745 } attrs { promote: false } } peers { permanent_uuid: "d8fa9a5917714233bef295f611c589ab" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 38331 } attrs { promote: false } } }
I20260926 06:57:42.321426 8720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fa9a5917714233bef295f611c589ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:57:42.325928 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.322972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44673 (local address 127.0.0.1:42388)
0926 06:57:42.323042 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.323051 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:42.323088 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:42.323464 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:42.323465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:42.323467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:42.323501 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.323502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.324370 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.324371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.325092 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.325094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.325745 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.325746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.325811 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.325815 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:42.325850 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 06:57:42.325881 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":2}
I20260926 06:57:42.326193 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.323204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.1.74.65:44673)
0926 06:57:42.323330 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:42.323332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:42.323339 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:42.323364 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:42.323365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:42.323376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:42.323557 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:42.323592 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.324280 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.325183 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.325655 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.326054 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.326071 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:42.326090 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 06:57:42.326137 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260926 06:57:42.326396 8314 tablet_service.cc:1467] Processing CreateTablet for tablet b9bdd7071f914a95bd13f5055747c7e1 (DEFAULT_TABLE table=test-table [id=d8e317b3443d479d839fb024d96c076c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:42.326515 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.322993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:42713 (local address 127.0.0.1:33986)
0926 06:57:42.323124 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.323134 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:42.323194 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:42.323436 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:42.323437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:42.323439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:42.323484 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.323486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.324278 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.324279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.325547 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.325548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.326276 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.326277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.326340 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.326345 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:57:42.326401 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:57:42.326440 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":81,"thread_start_us":90,"threads_started":1}
I20260926 06:57:42.326517 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.323188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.1.74.66:42713)
0926 06:57:42.323300 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:42.323302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:42.323310 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:42.323333 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:42.323333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:42.323343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:42.323432 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:42.323536 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.324193 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.325628 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.326188 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.326268 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.326282 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:42.326408 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 06:57:42.326452 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":48,"threads_started":1}
I20260926 06:57:42.326546 8314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9bdd7071f914a95bd13f5055747c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:42.326843 8451 tablet_service.cc:1467] Processing CreateTablet for tablet b9bdd7071f914a95bd13f5055747c7e1 (DEFAULT_TABLE table=test-table [id=d8e317b3443d479d839fb024d96c076c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:42.326977 8451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9bdd7071f914a95bd13f5055747c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:42.327384 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.322901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:39347 (local address 127.0.0.1:44648)
0926 06:57:42.322929 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.322940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:42.322976 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:42.323244 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:42.323245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:42.323247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:42.323318 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.323320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.324315 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.324316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.325498 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:42.325500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.327132 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.327133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:42.327196 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.327201 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:57:42.327293 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 06:57:42.327329 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":63}
I20260926 06:57:42.327440 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.322977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.1.74.67:39347)
0926 06:57:42.323099 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:42.323101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:42.323110 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:42.323140 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:42.323141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:42.323154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:42.323389 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:42.323431 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.324216 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.325599 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.327005 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.327111 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.327123 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:42.327299 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 06:57:42.327346 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260926 06:57:42.328280 8741 tablet_bootstrap.cc:492] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68: Bootstrap starting.
I20260926 06:57:42.328529 8742 tablet_bootstrap.cc:492] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: Bootstrap starting.
I20260926 06:57:42.328742 8587 tablet_service.cc:1467] Processing CreateTablet for tablet b9bdd7071f914a95bd13f5055747c7e1 (DEFAULT_TABLE table=test-table [id=d8e317b3443d479d839fb024d96c076c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:42.328776 8741 tablet_bootstrap.cc:654] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:42.328858 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9bdd7071f914a95bd13f5055747c7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:42.329069 8742 tablet_bootstrap.cc:654] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:42.329355 8741 tablet_bootstrap.cc:492] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68: No bootstrap required, opened a new log
I20260926 06:57:42.329406 8741 ts_tablet_manager.cc:1397] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:42.329519 8741 raft_consensus.cc:348] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:42.329624 8741 raft_consensus.cc:374] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:42.329659 8741 raft_consensus.cc:729] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcbe55244ab41129e2ebf22ccfdac68, State: Initialized, Role: FOLLOWER
I20260926 06:57:42.329704 8741 consensus_queue.cc:260] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:42.329855 8741 ts_tablet_manager.cc:1428] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:42.330077 8742 tablet_bootstrap.cc:492] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: No bootstrap required, opened a new log
I20260926 06:57:42.330169 8742 ts_tablet_manager.cc:1397] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:42.330295 8747 tablet_bootstrap.cc:492] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: Bootstrap starting.
I20260926 06:57:42.330415 8382 heartbeater.cc:503] Master 127.1.74.125:40745 was elected leader, sending a full tablet report...
W20260926 06:57:42.330470 8383 tablet.cc:2369] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:42.330475 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.330025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.65:38817)
0926 06:57:42.330380 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.330417 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":76,"threads_started":1}
I20260926 06:57:42.330885 8747 tablet_bootstrap.cc:654] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:42.330948 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.330816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.65:55109)
0926 06:57:42.330902 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.330926 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 06:57:42.331164 8742 raft_consensus.cc:348] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:42.331279 8742 raft_consensus.cc:374] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:42.331316 8742 raft_consensus.cc:729] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59a69a04ee3450289da29995b78644c, State: Initialized, Role: FOLLOWER
I20260926 06:57:42.331393 8742 consensus_queue.cc:260] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:42.331485 8747 tablet_bootstrap.cc:492] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: No bootstrap required, opened a new log
I20260926 06:57:42.331540 8747 ts_tablet_manager.cc:1397] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:42.331719 8747 raft_consensus.cc:348] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:42.331808 8747 raft_consensus.cc:374] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:42.331836 8747 raft_consensus.cc:729] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e271855ce4c63bc91e29cae1041da, State: Initialized, Role: FOLLOWER
I20260926 06:57:42.331905 8747 consensus_queue.cc:260] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:42.332088 8747 ts_tablet_manager.cc:1428] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:42.331565 8742 ts_tablet_manager.cc:1428] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 06:57:42.332476 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.331702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.66:56387)
0926 06:57:42.331838 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.331875 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":84,"threads_started":1}
I20260926 06:57:42.332593 8655 heartbeater.cc:503] Master 127.1.74.125:40745 was elected leader, sending a full tablet report...
I20260926 06:57:42.332842 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.332374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.67:39243)
0926 06:57:42.332504 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.332606 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260926 06:57:42.332991 8519 heartbeater.cc:503] Master 127.1.74.125:40745 was elected leader, sending a full tablet report...
I20260926 06:57:42.333375 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.333178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.66:46821)
0926 06:57:42.333306 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.333339 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260926 06:57:42.351390 8656 tablet.cc:2369] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:42.524494 8745 raft_consensus.cc:482] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:42.524608 8745 raft_consensus.cc:504] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:42.524806 8745 leader_election.cc:288] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376e271855ce4c63bc91e29cae1041da (127.1.74.67:39347), c59a69a04ee3450289da29995b78644c (127.1.74.66:42713)
I20260926 06:57:42.525100 8607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bdd7071f914a95bd13f5055747c7e1" candidate_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e271855ce4c63bc91e29cae1041da" is_pre_election: true
I20260926 06:57:42.525182 8607 raft_consensus.cc:2454] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bcbe55244ab41129e2ebf22ccfdac68 in term 0.
I20260926 06:57:42.525100 8471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bdd7071f914a95bd13f5055747c7e1" candidate_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59a69a04ee3450289da29995b78644c" is_pre_election: true
I20260926 06:57:42.525231 8471 raft_consensus.cc:2454] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bcbe55244ab41129e2ebf22ccfdac68 in term 0.
I20260926 06:57:42.525363 8268 leader_election.cc:302] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcbe55244ab41129e2ebf22ccfdac68, 376e271855ce4c63bc91e29cae1041da; no voters:
I20260926 06:57:42.525501 8745 raft_consensus.cc:2781] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:42.525556 8745 raft_consensus.cc:482] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:42.525584 8745 raft_consensus.cc:3037] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:42.526322 8745 raft_consensus.cc:504] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:42.526484 8745 leader_election.cc:288] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [CANDIDATE]: Term 1 election: Requested vote from peers 376e271855ce4c63bc91e29cae1041da (127.1.74.67:39347), c59a69a04ee3450289da29995b78644c (127.1.74.66:42713)
I20260926 06:57:42.526656 8607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bdd7071f914a95bd13f5055747c7e1" candidate_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e271855ce4c63bc91e29cae1041da"
I20260926 06:57:42.526674 8471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bdd7071f914a95bd13f5055747c7e1" candidate_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59a69a04ee3450289da29995b78644c"
I20260926 06:57:42.526743 8607 raft_consensus.cc:3037] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:42.526742 8471 raft_consensus.cc:3037] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:42.527521 8471 raft_consensus.cc:2454] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bcbe55244ab41129e2ebf22ccfdac68 in term 1.
I20260926 06:57:42.527521 8607 raft_consensus.cc:2454] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bcbe55244ab41129e2ebf22ccfdac68 in term 1.
I20260926 06:57:42.527729 8268 leader_election.cc:302] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcbe55244ab41129e2ebf22ccfdac68, c59a69a04ee3450289da29995b78644c; no voters:
I20260926 06:57:42.527828 8745 raft_consensus.cc:2781] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:42.527987 8745 raft_consensus.cc:686] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 LEADER]: Becoming Leader. State: Replica: 0bcbe55244ab41129e2ebf22ccfdac68, State: Running, Role: LEADER
I20260926 06:57:42.528034 8745 consensus_queue.cc:237] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } }
I20260926 06:57:42.528396 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.528251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.65:40271)
0926 06:57:42.528326 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.528360 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 06:57:42.528680 8014 catalog_manager.cc:5495] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bcbe55244ab41129e2ebf22ccfdac68 (127.1.74.65). New cstate: current_term: 1 leader_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bcbe55244ab41129e2ebf22ccfdac68" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 } health_report { overall_health: UNKNOWN } } }
I20260926 06:57:42.552449 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.550099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.1.74.65:44673)
0926 06:57:42.550139 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:42.550141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:42.550150 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:42.550223 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:42.550224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:42.550236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:42.550281 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:42.550390 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.550949 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.551864 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:42.552046 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:42.552100 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:42.552222 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:42.552246 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:42.552279 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:42.552332 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 06:57:42.552349 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:57:42.553481 8471 raft_consensus.cc:1261] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [term 1 FOLLOWER]: Refusing update from remote peer 0bcbe55244ab41129e2ebf22ccfdac68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:42.553689 8607 raft_consensus.cc:1261] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [term 1 FOLLOWER]: Refusing update from remote peer 0bcbe55244ab41129e2ebf22ccfdac68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:42.553697 8745 consensus_queue.cc:1035] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59a69a04ee3450289da29995b78644c" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:42.553944 8745 consensus_queue.cc:1035] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e271855ce4c63bc91e29cae1041da" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:42.554570 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.554434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.66:37295)
0926 06:57:42.554508 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.554538 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 06:57:42.555183 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.555070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.67:58611)
0926 06:57:42.555110 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.555146 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 06:57:42.555441 8757 mvcc.cc:205] Tried to move back new op lower bound from 7333502413016944640 to 7333502412915220480. Current Snapshot: MvccSnapshot[applied={T|T < 7333502413016944640}]
I20260926 06:57:42.555635 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.555541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.65:46227)
0926 06:57:42.555578 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.555603 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 06:57:42.555996 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.555844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.65:37117)
0926 06:57:42.555935 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.555964 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 06:57:42.556597 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.556481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.67:59669)
0926 06:57:42.556538 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.556569 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 06:57:42.556962 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.556870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44973 (local address 127.1.74.66:40559)
0926 06:57:42.556903 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.556929 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 06:57:42.557888 8012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:42.557971 8012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:42.559192 8012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c030d44f0c72420495cd8c27f9b508ae: Sending DeleteTablet for 3 replicas of tablet b9bdd7071f914a95bd13f5055747c7e1
I20260926 06:57:42.559533 8587 tablet_service.cc:1514] Processing DeleteTablet for tablet b9bdd7071f914a95bd13f5055747c7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:42 UTC) from {username='oryx', principal='oryx/127.1.74.125@KRBTEST.COM'} at 127.0.0.1:44648
I20260926 06:57:42.559612 8314 tablet_service.cc:1514] Processing DeleteTablet for tablet b9bdd7071f914a95bd13f5055747c7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:42 UTC) from {username='oryx', principal='oryx/127.1.74.125@KRBTEST.COM'} at 127.0.0.1:42388
I20260926 06:57:42.559722 8767 tablet_replica.cc:330] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: stopping tablet replica
I20260926 06:57:42.559778 8767 raft_consensus.cc:2229] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:42.559788 8451 tablet_service.cc:1514] Processing DeleteTablet for tablet b9bdd7071f914a95bd13f5055747c7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:42 UTC) from {username='oryx', principal='oryx/127.1.74.125@KRBTEST.COM'} at 127.0.0.1:33986
I20260926 06:57:42.559814 8768 tablet_replica.cc:330] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68: stopping tablet replica
I20260926 06:57:42.559826 8767 raft_consensus.cc:2258] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:42.559974 8769 tablet_replica.cc:330] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: stopping tablet replica
I20260926 06:57:42.559865 8768 raft_consensus.cc:2229] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:42.560099 8767 ts_tablet_manager.cc:1905] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:42.560067 8769 raft_consensus.cc:2229] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:42.560112 8768 raft_consensus.cc:2258] T b9bdd7071f914a95bd13f5055747c7e1 P 0bcbe55244ab41129e2ebf22ccfdac68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:42.560113 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 8163
I20260926 06:57:42.560220 8769 raft_consensus.cc:2258] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:42.560873 8769 ts_tablet_manager.cc:1905] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:42.562381 8767 ts_tablet_manager.cc:1918] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:42.562466 8767 log.cc:1196] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/b9bdd7071f914a95bd13f5055747c7e1
I20260926 06:57:42.562523 8769 ts_tablet_manager.cc:1918] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:42.562580 8769 log.cc:1196] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/b9bdd7071f914a95bd13f5055747c7e1
I20260926 06:57:42.562718 8767 ts_tablet_manager.cc:1939] T b9bdd7071f914a95bd13f5055747c7e1 P 376e271855ce4c63bc91e29cae1041da: Deleting consensus metadata
I20260926 06:57:42.562811 8769 ts_tablet_manager.cc:1939] T b9bdd7071f914a95bd13f5055747c7e1 P c59a69a04ee3450289da29995b78644c: Deleting consensus metadata
I20260926 06:57:42.563024 7999 catalog_manager.cc:4841] TS 376e271855ce4c63bc91e29cae1041da (127.1.74.67:39347): tablet b9bdd7071f914a95bd13f5055747c7e1 (table test-table [id=d8e317b3443d479d839fb024d96c076c]) successfully deleted
I20260926 06:57:42.563129 7999 catalog_manager.cc:4841] TS c59a69a04ee3450289da29995b78644c (127.1.74.66:42713): tablet b9bdd7071f914a95bd13f5055747c7e1 (table test-table [id=d8e317b3443d479d839fb024d96c076c]) successfully deleted
I20260926 06:57:42.567998 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:42.567878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44673 (local address 127.0.0.1:42398)
0926 06:57:42.567933 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:42.567959 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 06:57:42.568082 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 8388
W20260926 06:57:42.568169 8001 catalog_manager.cc:4584] TS 0bcbe55244ab41129e2ebf22ccfdac68 (127.1.74.65:44673): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9bdd7071f914a95bd13f5055747c7e1: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44673: connect: Connection refused (error 111)
I20260926 06:57:42.573069 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 8524
I20260926 06:57:42.577750 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 7880
I20260926 06:57:42.582593 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 8068
2026-09-26T06:57:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4710 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:42 dist-test-slave-sfvt krb5kdc[8772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:42 dist-test-slave-sfvt krb5kdc[8772](info): set up 2 sockets
Sep 26 06:57:42 dist-test-slave-sfvt krb5kdc[8772](info): commencing operation
krb5kdc: starting...
W20260926 06:57:43.623929 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:43 dist-test-slave-sfvt krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.126.keytab.
Entry for principal oryx/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.126.keytab.
I20260926 06:57:43.742051 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:40563
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.126.keytab
--principal=oryx/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:40563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:43.821753 8785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:43.822005 8785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:43.822041 8785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:43.823163 8785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:43.823239 8785 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 06:57:43.823271 8785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:43.823334 8785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:43.823369 8785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:43.823396 8785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:43.824920 8785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:40563
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.126.keytab
--principal=oryx/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:40563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:43.825201 8785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:43.825423 8785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:43.830374 8790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:43.830405 8785 server_base.cc:1034] running on GCE node
W20260926 06:57:43.830402 8793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:43.830376 8791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:43.830982 8785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:43.831321 8785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:43.832485 8785 hybrid_clock.cc:648] HybridClock initialized: now 1790405863832474 us; error 29 us; skew 500 ppm
Sep 26 06:57:43 dist-test-slave-sfvt krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405863, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:43.834939 8785 init.cc:377] Logged in from keytab as oryx/127.1.74.126@KRBTEST.COM (short username oryx)
I20260926 06:57:43.835429 8785 webserver.cc:460] Webserver started at http://127.1.74.126:38053/ using document root <none> and password file <none>
I20260926 06:57:43.835671 8785 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:43.835739 8785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:43.835891 8785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:43.836951 8785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "9fe394fb1cc443cd87694a36ac43957e"
format_stamp: "Formatted at 2026-09-26 06:57:43 on dist-test-slave-sfvt"
server_key: "4daf03194c8dfe13811d1a0aed164909"
server_key_iv: "8b33e79c04e735e6467a09322f59ebae"
server_key_version: "encryptionkey@0"
I20260926 06:57:43.837304 8785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "9fe394fb1cc443cd87694a36ac43957e"
format_stamp: "Formatted at 2026-09-26 06:57:43 on dist-test-slave-sfvt"
server_key: "4daf03194c8dfe13811d1a0aed164909"
server_key_iv: "8b33e79c04e735e6467a09322f59ebae"
server_key_version: "encryptionkey@0"
I20260926 06:57:43.838629 8785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:57:43.839449 8800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:43.839742 8785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:57:43.839848 8785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "9fe394fb1cc443cd87694a36ac43957e"
format_stamp: "Formatted at 2026-09-26 06:57:43 on dist-test-slave-sfvt"
server_key: "4daf03194c8dfe13811d1a0aed164909"
server_key_iv: "8b33e79c04e735e6467a09322f59ebae"
server_key_version: "encryptionkey@0"
I20260926 06:57:43.839922 8785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:43.856128 8785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:43.858247 8785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:43.858397 8785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:43.862773 8785 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:40563
I20260926 06:57:43.862913 8862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:40563 every 8 connection(s)
I20260926 06:57:43.863214 8785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:43.863936 8863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:43.866222 8863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e: Bootstrap starting.
I20260926 06:57:43.866580 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 8785
I20260926 06:57:43.866684 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:43.866863 1321 external_mini_cluster.cc:1428] Setting key 6785293366a7d439ab373020c73c6323
I20260926 06:57:43.866911 8863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:43.867250 8863 log.cc:824] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:43.868034 8863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e: No bootstrap required, opened a new log
I20260926 06:57:43.869771 8863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe394fb1cc443cd87694a36ac43957e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40563 } }
I20260926 06:57:43.870021 8863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:43.870096 8863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe394fb1cc443cd87694a36ac43957e, State: Initialized, Role: FOLLOWER
I20260926 06:57:43.870245 8863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe394fb1cc443cd87694a36ac43957e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40563 } }
I20260926 06:57:43.870338 8863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:43.870383 8863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:43.870432 8863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:43.871172 8863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe394fb1cc443cd87694a36ac43957e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40563 } }
I20260926 06:57:43.871310 8863 leader_election.cc:302] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fe394fb1cc443cd87694a36ac43957e; no voters:
I20260926 06:57:43.871479 8863 leader_election.cc:288] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:43.871502 8868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:43.871621 8868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [term 1 LEADER]: Becoming Leader. State: Replica: 9fe394fb1cc443cd87694a36ac43957e, State: Running, Role: LEADER
I20260926 06:57:43.871768 8863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:43.871776 8868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe394fb1cc443cd87694a36ac43957e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40563 } }
I20260926 06:57:43.872184 8870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fe394fb1cc443cd87694a36ac43957e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe394fb1cc443cd87694a36ac43957e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40563 } } }
I20260926 06:57:43.872522 8870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:43.872447 8869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fe394fb1cc443cd87694a36ac43957e. Latest consensus state: current_term: 1 leader_uuid: "9fe394fb1cc443cd87694a36ac43957e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fe394fb1cc443cd87694a36ac43957e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 40563 } } }
I20260926 06:57:43.872653 8869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:43.873025 8878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:43.873525 8878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 06:57:43 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:43.875092 8878 catalog_manager.cc:1347] Generated new cluster ID: 42a0850c66b24a4994503c774ca4c436
I20260926 06:57:43.875149 8878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:43.878111 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:43.867393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.1.74.126:40563)
0926 06:57:43.867552 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:43.867561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:43.867599 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:57:43.868016 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:43.868022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:43.868068 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:43.868237 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:43.868998 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:43.870302 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:43.871456 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:43.872077 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:43.872536 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:43.874908 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:43.874912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:43.874913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:43.874925 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:43.876554 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:43.876929 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:43.876932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:43.876932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:43.877030 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:43.877245 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:43.877246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:43.877246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:43.877487 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:43.877637 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:43.877752 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 06:57:43.877889 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260926 06:57:43.879355 8878 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:43.880062 8878 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:43.895946 8878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e: Generated new TSK 0
I20260926 06:57:43.896183 8878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab.
Entry for principal oryx/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab.
I20260926 06:57:43.924436 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:40563
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab
--principal=oryx/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:44.003317 8889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:44.003556 8889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:44.003595 8889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:44.004913 8889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:44.004963 8889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:44.004997 8889 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:57:44.005034 8889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:44.007148 8889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab
--principal=oryx/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.74.126:40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:44.007433 8889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:44.007676 8889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:44.012573 8897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:44.012626 8895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:44.012725 8889 server_base.cc:1034] running on GCE node
W20260926 06:57:44.012562 8894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:44.013027 8889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:44.013219 8889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:44.014360 8889 hybrid_clock.cc:648] HybridClock initialized: now 1790405864014344 us; error 31 us; skew 500 ppm
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:44.016919 8889 init.cc:377] Logged in from keytab as oryx/127.1.74.65@KRBTEST.COM (short username oryx)
I20260926 06:57:44.017386 8889 webserver.cc:460] Webserver started at http://127.1.74.65:46437/ using document root <none> and password file <none>
I20260926 06:57:44.017603 8889 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:44.017647 8889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:44.017787 8889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:44.018833 8889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "b953454746c6422699d2bdff3fa11138"
format_stamp: "Formatted at 2026-09-26 06:57:44 on dist-test-slave-sfvt"
server_key: "441195c97fbf5d992b31dd0d68bee56a"
server_key_iv: "ed8b78ea54a6b4ba1dafa6bf62754bf2"
server_key_version: "encryptionkey@0"
I20260926 06:57:44.019192 8889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "b953454746c6422699d2bdff3fa11138"
format_stamp: "Formatted at 2026-09-26 06:57:44 on dist-test-slave-sfvt"
server_key: "441195c97fbf5d992b31dd0d68bee56a"
server_key_iv: "ed8b78ea54a6b4ba1dafa6bf62754bf2"
server_key_version: "encryptionkey@0"
I20260926 06:57:44.020535 8889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:44.021291 8904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.021489 8889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:44.021570 8889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "b953454746c6422699d2bdff3fa11138"
format_stamp: "Formatted at 2026-09-26 06:57:44 on dist-test-slave-sfvt"
server_key: "441195c97fbf5d992b31dd0d68bee56a"
server_key_iv: "ed8b78ea54a6b4ba1dafa6bf62754bf2"
server_key_version: "encryptionkey@0"
I20260926 06:57:44.021622 8889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:44.030942 8889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:44.032826 8889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:44.032929 8889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:44.033360 8889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:44.033401 8889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.033422 8889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:44.033435 8889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.039870 8889 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:43627
I20260926 06:57:44.039918 9019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:43627 every 8 connection(s)
I20260926 06:57:44.040221 8889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:44.043542 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.033821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:56963 (local address 127.1.74.126:40563)
0926 06:57:44.033894 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.033897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.034054 (+ 157us) server_negotiation.cc:407] Connection header received
0926 06:57:44.034517 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.034518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.034532 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.034619 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.035501 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.036330 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.037913 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.038248 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.038341 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.040872 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:44.040873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:44.040873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:44.040880 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:44.042160 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.042567 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.042569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.042569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.042633 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.042887 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.042888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.042888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.043020 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:44.043096 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.043388 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.043474 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:57:44.043723 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.033772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:40563 (local address 127.1.74.65:56963)
0926 06:57:44.033927 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.033966 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.034427 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.034666 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.034669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.035052 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:44.035398 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.035405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.036424 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.036425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.037816 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.037820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.038342 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.038343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.038447 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.039003 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:44.039014 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:44.040782 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:44.042232 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.042238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.042239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.042469 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.042710 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.042711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.042711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.042805 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.043089 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:44.043093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:44.043198 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.043365 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.043488 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260926 06:57:44.044330 9020 heartbeater.cc:347] Connected to a master server at 127.1.74.126:40563
I20260926 06:57:44.044437 9020 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:44.044606 9020 heartbeater.cc:511] Master 127.1.74.126:40563 requested a full tablet report, sending...
I20260926 06:57:44.045209 8816 ts_manager.cc:194] Registered new tserver with Master: b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627)
I20260926 06:57:44.046069 8816 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:56963
I20260926 06:57:44.049342 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 8889
I20260926 06:57:44.049461 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:44.049655 1321 external_mini_cluster.cc:1428] Setting key 6e3bbfe3559577b3011bf7274294cf40
I20260926 06:57:44.051044 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.045454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:45359 (local address 127.1.74.126:40563)
0926 06:57:44.045490 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.045492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.045604 (+ 112us) server_negotiation.cc:407] Connection header received
0926 06:57:44.045628 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.045629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.045638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.045704 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.045965 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.046597 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.047454 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.047633 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.047723 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.048482 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:44.048483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:44.048483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:44.048489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:44.050032 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.050323 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.050324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.050325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.050380 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.050601 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.050602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.050602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.050717 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:44.050770 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.050929 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.050987 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:57:44.051446 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.045379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:40563 (local address 127.1.74.65:45359)
0926 06:57:44.045524 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.045534 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.045587 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.045715 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.045716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.045854 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:44.045904 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.045906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.046687 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.046688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.047366 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.047369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.047736 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.047737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.047781 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.048143 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:44.048147 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:44.048410 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:44.050104 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.050105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.050105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.050254 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.050446 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.050447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.050449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.050519 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.050784 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:44.050785 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:44.050832 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.050915 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.051029 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for oryx/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab.
Entry for principal oryx/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab.
I20260926 06:57:44.074577 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:40563
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab
--principal=oryx/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:44.154644 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:44.154858 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:44.154882 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:44.156097 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:44.156144 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:44.156165 9029 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:57:44.156183 9029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:44.157719 9029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab
--principal=oryx/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.74.126:40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:44.157984 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:44.158213 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:44.163523 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:44.163523 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:44.163535 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:44.163590 9029 server_base.cc:1034] running on GCE node
I20260926 06:57:44.163957 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:44.164206 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:44.165373 9029 hybrid_clock.cc:648] HybridClock initialized: now 1790405864165355 us; error 35 us; skew 500 ppm
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:44.167876 9029 init.cc:377] Logged in from keytab as oryx/127.1.74.66@KRBTEST.COM (short username oryx)
I20260926 06:57:44.168375 9029 webserver.cc:460] Webserver started at http://127.1.74.66:33687/ using document root <none> and password file <none>
I20260926 06:57:44.168623 9029 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:44.168700 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:44.168848 9029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:44.169946 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "f6e48a24c52c47cd9272e242365fe560"
format_stamp: "Formatted at 2026-09-26 06:57:44 on dist-test-slave-sfvt"
server_key: "0ea78628609ba147c865cc3851ba8a8a"
server_key_iv: "14fa5383a8cfa0b7d555f9fd0e09a9fc"
server_key_version: "encryptionkey@0"
I20260926 06:57:44.170301 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "f6e48a24c52c47cd9272e242365fe560"
format_stamp: "Formatted at 2026-09-26 06:57:44 on dist-test-slave-sfvt"
server_key: "0ea78628609ba147c865cc3851ba8a8a"
server_key_iv: "14fa5383a8cfa0b7d555f9fd0e09a9fc"
server_key_version: "encryptionkey@0"
I20260926 06:57:44.171718 9029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:44.172564 9044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.172852 9029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:44.172992 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "f6e48a24c52c47cd9272e242365fe560"
format_stamp: "Formatted at 2026-09-26 06:57:44 on dist-test-slave-sfvt"
server_key: "0ea78628609ba147c865cc3851ba8a8a"
server_key_iv: "14fa5383a8cfa0b7d555f9fd0e09a9fc"
server_key_version: "encryptionkey@0"
I20260926 06:57:44.173072 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:44.187707 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:44.189970 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:44.190182 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:44.190778 9029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:44.190867 9029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.190919 9029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:44.190948 9029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:44.198400 9029 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:38597
I20260926 06:57:44.198462 9159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:38597 every 8 connection(s)
I20260926 06:57:44.198940 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:44.199360 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 9029
I20260926 06:57:44.199465 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:44.199640 1321 external_mini_cluster.cc:1428] Setting key 248dac024ab18b6de24fe6127b90a0a0
I20260926 06:57:44.200955 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.191175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:40563 (local address 127.1.74.66:41551)
0926 06:57:44.191350 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.191390 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.191914 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.192174 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.192177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.192427 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:44.192640 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.192647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.193347 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.193348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.194165 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.194167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.194591 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.194592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.194761 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.195282 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:44.195293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:44.197592 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:44.199564 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.199570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.199572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.199875 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.200070 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.200071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.200071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.200197 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.200466 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:44.200471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:44.200582 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.200661 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.200806 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260926 06:57:44.200953 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.191166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:41551 (local address 127.1.74.126:40563)
0926 06:57:44.191243 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.191245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.191497 (+ 252us) server_negotiation.cc:407] Connection header received
0926 06:57:44.192016 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.192018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.192032 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.192120 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.192746 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.193245 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.194264 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.194500 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.194573 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.197708 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:44.197710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:44.197710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:44.197717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:44.199474 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.199946 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.199947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.199948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.200010 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.200274 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.200275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.200276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.200400 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:44.200464 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.200726 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.200812 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 06:57:44.201450 9160 heartbeater.cc:347] Connected to a master server at 127.1.74.126:40563
I20260926 06:57:44.201525 9160 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:44.201705 9160 heartbeater.cc:511] Master 127.1.74.126:40563 requested a full tablet report, sending...
I20260926 06:57:44.202518 8817 ts_manager.cc:194] Registered new tserver with Master: f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597)
I20260926 06:57:44.202980 8817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:41551
WARNING: no policy specified for oryx/127.1.74.67@KRBTEST.COM; defaulting to no policy
I20260926 06:57:44.208891 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.202472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:38727 (local address 127.1.74.126:40563)
0926 06:57:44.202512 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.202514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.202638 (+ 124us) server_negotiation.cc:407] Connection header received
0926 06:57:44.202718 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.202719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.202728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.202785 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.203125 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.203955 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.205066 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.205251 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.205321 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.206344 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:44.206345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:44.206346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:44.206351 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:44.207741 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.208142 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.208142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.208143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.208188 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.208377 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.208377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.208377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.208509 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:44.208560 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.208755 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.208831 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:57:44.209156 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.202411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:40563 (local address 127.1.74.66:38727)
0926 06:57:44.202568 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.202580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.202661 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.202788 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.202789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.202977 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:44.203042 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.203044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.204068 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.204069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.204986 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.204990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.205342 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.205343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.205417 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.205824 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:44.205830 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:44.206255 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:44.207826 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.207827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.207827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.208066 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.208249 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.208250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.208250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.208317 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.208575 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:44.208576 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:44.208637 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.208719 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.208837 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab.
Entry for principal oryx/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab.
I20260926 06:57:44.226012 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:40563
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab
--principal=oryx/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:44.306907 9169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:44.307114 9169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:44.307138 9169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:44.308308 9169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:44.308350 9169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:44.308370 9169 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:57:44.308429 9169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:44.309957 9169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab
--principal=oryx/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.74.126:40563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:44.310204 9169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:44.310437 9169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:44.315522 9174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:44.315562 9175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:44.315538 9177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:44.315749 9169 server_base.cc:1034] running on GCE node
I20260926 06:57:44.316002 9169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:44.316184 9169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:44.317373 9169 hybrid_clock.cc:648] HybridClock initialized: now 1790405864317362 us; error 40 us; skew 500 ppm
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:44.319932 9169 init.cc:377] Logged in from keytab as oryx/127.1.74.67@KRBTEST.COM (short username oryx)
I20260926 06:57:44.320384 9169 webserver.cc:460] Webserver started at http://127.1.74.67:37803/ using document root <none> and password file <none>
I20260926 06:57:44.320575 9169 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:44.320614 9169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:44.320699 9169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:44.321729 9169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "9d308dfc112d4943940a9c226e275123"
format_stamp: "Formatted at 2026-09-26 06:57:44 on dist-test-slave-sfvt"
server_key: "6498affecfa7932e4f206e477585c0c6"
server_key_iv: "ee32ee0c819dfe3648e262bdc7e7e427"
server_key_version: "encryptionkey@0"
I20260926 06:57:44.322027 9169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "9d308dfc112d4943940a9c226e275123"
format_stamp: "Formatted at 2026-09-26 06:57:44 on dist-test-slave-sfvt"
server_key: "6498affecfa7932e4f206e477585c0c6"
server_key_iv: "ee32ee0c819dfe3648e262bdc7e7e427"
server_key_version: "encryptionkey@0"
I20260926 06:57:44.323251 9169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:57:44.324100 9184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.324249 9169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:44.324330 9169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "9d308dfc112d4943940a9c226e275123"
format_stamp: "Formatted at 2026-09-26 06:57:44 on dist-test-slave-sfvt"
server_key: "6498affecfa7932e4f206e477585c0c6"
server_key_iv: "ee32ee0c819dfe3648e262bdc7e7e427"
server_key_version: "encryptionkey@0"
I20260926 06:57:44.324386 9169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:44.337755 9169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:44.339632 9169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:44.339740 9169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:44.340166 9169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:44.340204 9169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.340227 9169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:44.340267 9169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.346894 9169 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:39163
I20260926 06:57:44.346958 9299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:39163 every 8 connection(s)
I20260926 06:57:44.347321 9169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:44.350216 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.340659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:58891 (local address 127.1.74.126:40563)
0926 06:57:44.340750 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.340752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.340996 (+ 244us) server_negotiation.cc:407] Connection header received
0926 06:57:44.341471 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.341472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.341486 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.341565 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.342431 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.343367 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.344926 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.345241 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.345327 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.347906 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:44.347908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:44.347908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:44.347916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:44.349126 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.349496 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.349497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.349497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.349544 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.349740 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.349740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.349741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.349834 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:44.349882 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.350097 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.350154 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 06:57:44.350425 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.340649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:40563 (local address 127.1.74.67:58891)
0926 06:57:44.340838 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.340877 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.341356 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.341648 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.341652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.341990 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:44.342274 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.342281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.343467 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.343468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.344836 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.344838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.345326 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.345327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.345426 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.345983 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:44.345994 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:44.347796 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:44.349195 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.349200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.349202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.349428 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.349598 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.349599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.349599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.349688 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.349884 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:44.349888 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:44.349992 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.350071 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.350181 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":120,"threads_started":1}
I20260926 06:57:44.350744 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 9169
I20260926 06:57:44.350862 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:44.350873 9300 heartbeater.cc:347] Connected to a master server at 127.1.74.126:40563
I20260926 06:57:44.350953 9300 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:44.351071 1321 external_mini_cluster.cc:1428] Setting key 4eb285d4e58db904650a446d5fafeaec
I20260926 06:57:44.351158 9300 heartbeater.cc:511] Master 127.1.74.126:40563 requested a full tablet report, sending...
I20260926 06:57:44.351541 8817 ts_manager.cc:194] Registered new tserver with Master: 9d308dfc112d4943940a9c226e275123 (127.1.74.67:39163)
I20260926 06:57:44.352003 8817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:58891
I20260926 06:57:44.352211 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:57:44.358106 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.352011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:41811 (local address 127.1.74.126:40563)
0926 06:57:44.352188 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.352191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.352198 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:44.352215 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.352216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.352225 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.352278 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.352619 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.353130 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.354171 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.354472 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.354547 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.355455 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:44.355456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:44.355456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:44.355463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:44.357023 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.357445 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.357446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.357446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.357496 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.357695 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.357695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.357696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.357785 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:44.357836 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.357993 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.358048 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260926 06:57:44.358251 9306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.351937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:40563 (local address 127.1.74.67:41811)
0926 06:57:44.352074 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.352084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.352150 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.352290 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.352291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.352471 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:44.352534 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.352535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.353234 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.353235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.354086 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.354088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.354549 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.354550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.354594 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.355045 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:44.355050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:44.355380 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:44.357108 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.357109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.357110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.357366 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.357562 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.357563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.357564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.357634 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.357845 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:44.357846 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:44.357892 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.357976 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.358088 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:57:44.374248 8866 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:44.369003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.1.74.126:40563)
0926 06:57:44.369069 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.369071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.369096 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:57:44.369125 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.369126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.369134 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.369175 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.369709 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.370175 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.370893 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.371070 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.371117 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.374189 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46476: BlockingRecv error: recv got EOF from 127.0.0.1:46476 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:44.382551 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.375125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.1.74.126:40563)
0926 06:57:44.375200 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.375202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.375281 (+ 79us) server_negotiation.cc:407] Connection header received
0926 06:57:44.375386 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.375387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.375397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.375466 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.376057 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.376529 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.377282 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.377469 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.377553 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.379658 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:44.379659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:44.379660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:44.379666 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:44.381104 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.381537 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.381539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.381539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.381613 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.381830 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.381831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.381831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.381968 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:44.382041 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.382321 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.382425 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 06:57:44.387943 8817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:44.390259 8817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:44.396440 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.392624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:43627 (local address 127.0.0.1:57674)
0926 06:57:44.392827 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.392838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.392885 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.393092 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.393095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.393097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:44.393148 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.393149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.394396 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.394397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.395344 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.395346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.396123 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.396124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.396212 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.396216 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.396265 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.396328 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":30,"threads_started":1}
I20260926 06:57:44.396450 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.392650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.1.74.67:39163)
0926 06:57:44.392815 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.392821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.392832 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:57:44.392938 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.392941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.392961 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.393015 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:44.393299 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.394041 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.395297 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.396004 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.396037 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.396083 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.396291 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.396330 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260926 06:57:44.396445 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.392446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:39163 (local address 127.0.0.1:36514)
0926 06:57:44.392777 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.392787 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.392838 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.393110 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.393111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.393113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:44.393164 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.393165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.394127 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.394128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.395170 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.395172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.396127 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.396128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.396180 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.396189 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.396267 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.396333 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":24,"thread_start_us":48,"threads_started":1}
I20260926 06:57:44.396775 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.392781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.1.74.65:43627)
0926 06:57:44.392938 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.392942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.392950 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:44.392986 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.392988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.393002 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.393075 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:44.393199 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.394304 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.395438 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.395953 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.396018 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.396089 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.396632 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.396681 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260926 06:57:44.397172 9234 tablet_service.cc:1467] Processing CreateTablet for tablet d943db3aacf144cfa64d1b3316b4fcaa (DEFAULT_TABLE table=test-table [id=9b104da7241d41b29b571e6d7d4b3b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:44.397323 8954 tablet_service.cc:1467] Processing CreateTablet for tablet d943db3aacf144cfa64d1b3316b4fcaa (DEFAULT_TABLE table=test-table [id=9b104da7241d41b29b571e6d7d4b3b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:44.397528 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d943db3aacf144cfa64d1b3316b4fcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.397750 8954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d943db3aacf144cfa64d1b3316b4fcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.399904 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.392411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:38597 (local address 127.0.0.1:41722)
0926 06:57:44.392531 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.392571 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.392636 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.393555 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.393556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.393558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:44.393591 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.393592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.394747 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.394748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.395815 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.395818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.399608 (+ 3790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.399609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.399691 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.399697 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.399763 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.399813 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":75,"threads_started":1}
I20260926 06:57:44.400216 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.392541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.1.74.66:38597)
0926 06:57:44.393362 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.393366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.393375 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:44.393408 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.393410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.393424 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.393496 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:44.393658 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.394665 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.395904 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.396958 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.399476 (+ 2518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.399573 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.399794 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.399847 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":83,"threads_started":1}
I20260926 06:57:44.400697 9325 tablet_bootstrap.cc:492] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: Bootstrap starting.
I20260926 06:57:44.400813 9094 tablet_service.cc:1467] Processing CreateTablet for tablet d943db3aacf144cfa64d1b3316b4fcaa (DEFAULT_TABLE table=test-table [id=9b104da7241d41b29b571e6d7d4b3b9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:44.400997 9326 tablet_bootstrap.cc:492] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: Bootstrap starting.
I20260926 06:57:44.401130 9094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d943db3aacf144cfa64d1b3316b4fcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.401597 9325 tablet_bootstrap.cc:654] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.401649 9326 tablet_bootstrap.cc:654] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.401899 9325 log.cc:824] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:44.401957 9326 log.cc:824] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:44.402691 9326 tablet_bootstrap.cc:492] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: No bootstrap required, opened a new log
I20260926 06:57:44.402691 9325 tablet_bootstrap.cc:492] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: No bootstrap required, opened a new log
I20260926 06:57:44.402755 9326 ts_tablet_manager.cc:1397] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:44.402755 9325 ts_tablet_manager.cc:1397] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:57:44.404762 9325 raft_consensus.cc:348] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.404932 9325 raft_consensus.cc:374] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.404968 9325 raft_consensus.cc:729] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d308dfc112d4943940a9c226e275123, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.404899 9326 raft_consensus.cc:348] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.405045 9326 raft_consensus.cc:374] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.405074 9326 raft_consensus.cc:729] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b953454746c6422699d2bdff3fa11138, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.405084 9325 consensus_queue.cc:260] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.405169 9326 consensus_queue.cc:260] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.405342 9325 ts_tablet_manager.cc:1428] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:57:44.405395 9326 ts_tablet_manager.cc:1428] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:57:44.405496 9020 heartbeater.cc:503] Master 127.1.74.126:40563 was elected leader, sending a full tablet report...
I20260926 06:57:44.405745 9300 heartbeater.cc:503] Master 127.1.74.126:40563 was elected leader, sending a full tablet report...
I20260926 06:57:44.406620 9329 tablet_bootstrap.cc:492] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: Bootstrap starting.
I20260926 06:57:44.407333 9329 tablet_bootstrap.cc:654] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.407701 9329 log.cc:824] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:44.408372 9329 tablet_bootstrap.cc:492] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: No bootstrap required, opened a new log
I20260926 06:57:44.408422 9329 ts_tablet_manager.cc:1397] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:57:44.409915 9329 raft_consensus.cc:348] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.410036 9329 raft_consensus.cc:374] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.410058 9329 raft_consensus.cc:729] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e48a24c52c47cd9272e242365fe560, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.410200 9329 consensus_queue.cc:260] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.410455 9329 ts_tablet_manager.cc:1428] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:57:44.410509 9160 heartbeater.cc:503] Master 127.1.74.126:40563 was elected leader, sending a full tablet report...
W20260926 06:57:44.450343 9161 tablet.cc:2369] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:44.465746 9331 raft_consensus.cc:482] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:44.465870 9331 raft_consensus.cc:504] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.466311 9331 leader_election.cc:288] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597), 9d308dfc112d4943940a9c226e275123 (127.1.74.67:39163)
I20260926 06:57:44.470140 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.466545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:38597 (local address 127.1.74.65:34587)
0926 06:57:44.466596 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.466609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.466654 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.466831 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.466832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.466833 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:44.466911 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.466913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.467815 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.467816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.469078 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.469081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.469830 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.469831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.469917 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.469920 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.469983 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.470027 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":46}
I20260926 06:57:44.470139 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.466551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:39163 (local address 127.1.74.65:34359)
0926 06:57:44.466922 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.466931 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.467011 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.467144 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.467144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.467146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:44.467210 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.467211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.468048 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.468049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.469134 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.469136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.469830 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.469831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.469917 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.469921 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.469981 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.470027 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":161,"thread_start_us":139,"threads_started":1}
I20260926 06:57:44.470202 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.466618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:34359 (local address 127.1.74.67:39163)
0926 06:57:44.466669 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.466671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.467001 (+ 330us) server_negotiation.cc:407] Connection header received
0926 06:57:44.467072 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.467073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.467082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.467119 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:44.467310 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.467975 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.469212 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.469743 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.469822 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.469830 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.470084 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.470130 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:57:44.470202 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.466618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:34587 (local address 127.1.74.66:38597)
0926 06:57:44.466682 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.466685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.466701 (+ 16us) server_negotiation.cc:407] Connection header received
0926 06:57:44.466726 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.466727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.466736 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.466798 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:44.466996 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.467729 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.469198 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.469740 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.469820 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.469830 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.470084 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.470132 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:57:44.470582 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d943db3aacf144cfa64d1b3316b4fcaa" candidate_uuid: "b953454746c6422699d2bdff3fa11138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560" is_pre_election: true
I20260926 06:57:44.470628 9254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d943db3aacf144cfa64d1b3316b4fcaa" candidate_uuid: "b953454746c6422699d2bdff3fa11138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d308dfc112d4943940a9c226e275123" is_pre_election: true
I20260926 06:57:44.470772 9114 raft_consensus.cc:2454] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b953454746c6422699d2bdff3fa11138 in term 0.
I20260926 06:57:44.470795 9254 raft_consensus.cc:2454] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b953454746c6422699d2bdff3fa11138 in term 0.
I20260926 06:57:44.471004 8907 leader_election.cc:302] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b953454746c6422699d2bdff3fa11138, f6e48a24c52c47cd9272e242365fe560; no voters:
I20260926 06:57:44.471189 9331 raft_consensus.cc:2781] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:44.471271 9331 raft_consensus.cc:482] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:44.471290 9331 raft_consensus.cc:3037] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:44.472301 9331 raft_consensus.cc:504] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.472436 9331 leader_election.cc:288] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [CANDIDATE]: Term 1 election: Requested vote from peers f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597), 9d308dfc112d4943940a9c226e275123 (127.1.74.67:39163)
I20260926 06:57:44.472620 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d943db3aacf144cfa64d1b3316b4fcaa" candidate_uuid: "b953454746c6422699d2bdff3fa11138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560"
I20260926 06:57:44.472760 9114 raft_consensus.cc:3037] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:44.473019 9254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d943db3aacf144cfa64d1b3316b4fcaa" candidate_uuid: "b953454746c6422699d2bdff3fa11138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d308dfc112d4943940a9c226e275123"
I20260926 06:57:44.473121 9254 raft_consensus.cc:3037] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:44.473477 9114 raft_consensus.cc:2454] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b953454746c6422699d2bdff3fa11138 in term 1.
I20260926 06:57:44.473637 8907 leader_election.cc:302] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b953454746c6422699d2bdff3fa11138, f6e48a24c52c47cd9272e242365fe560; no voters:
I20260926 06:57:44.473737 9331 raft_consensus.cc:2781] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:44.473875 9331 raft_consensus.cc:686] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 1 LEADER]: Becoming Leader. State: Replica: b953454746c6422699d2bdff3fa11138, State: Running, Role: LEADER
I20260926 06:57:44.473968 9331 consensus_queue.cc:237] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.474027 9254 raft_consensus.cc:2454] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b953454746c6422699d2bdff3fa11138 in term 1.
I20260926 06:57:44.474870 8816 catalog_manager.cc:5495] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 reported cstate change: term changed from 0 to 1, leader changed from <none> to b953454746c6422699d2bdff3fa11138 (127.1.74.65). New cstate: current_term: 1 leader_uuid: "b953454746c6422699d2bdff3fa11138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } health_report { overall_health: UNKNOWN } } }
I20260926 06:57:44.490068 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.487987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.1.74.65:43627)
0926 06:57:44.488030 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.488032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.488040 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:44.488111 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.488112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.488128 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.488183 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:44.488274 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.488743 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.489472 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.489646 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.489696 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.489818 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:44.489910 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:44.489967 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.489997 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.490013 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:57:44.491833 9254 raft_consensus.cc:1261] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Refusing update from remote peer b953454746c6422699d2bdff3fa11138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:44.491860 9114 raft_consensus.cc:1261] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Refusing update from remote peer b953454746c6422699d2bdff3fa11138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:44.492209 9331 consensus_queue.cc:1035] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:44.492344 9335 consensus_queue.cc:1035] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:44.494757 9339 mvcc.cc:205] Tried to move back new op lower bound from 7333502420955389952 to 7333502420885721088. Current Snapshot: MvccSnapshot[applied={T|T < 7333502420955389952}]
I20260926 06:57:44.497135 8816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:44.497234 8816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:44.498008 8816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e: Sending DeleteTablet for 3 replicas of tablet d943db3aacf144cfa64d1b3316b4fcaa
I20260926 06:57:44.498343 9094 tablet_service.cc:1514] Processing DeleteTablet for tablet d943db3aacf144cfa64d1b3316b4fcaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:44 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:41722
I20260926 06:57:44.498373 8954 tablet_service.cc:1514] Processing DeleteTablet for tablet d943db3aacf144cfa64d1b3316b4fcaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:44 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57674
I20260926 06:57:44.498534 9234 tablet_service.cc:1514] Processing DeleteTablet for tablet d943db3aacf144cfa64d1b3316b4fcaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:44 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:36514
I20260926 06:57:44.498672 8816 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:44.498839 8816 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:44.498970 9348 tablet_replica.cc:330] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: stopping tablet replica
I20260926 06:57:44.499068 9348 raft_consensus.cc:2229] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:44.499183 9348 raft_consensus.cc:2258] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:44.499238 9350 tablet_replica.cc:330] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: stopping tablet replica
I20260926 06:57:44.499588 9350 raft_consensus.cc:2229] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:44.499825 9350 raft_consensus.cc:2258] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:44.500017 8954 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1e3635139e495fbb9f9cd9e2115046 (DEFAULT_TABLE table=test-table [id=907dc575fbab48bb86f8272d96fda2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:44.500151 8954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1e3635139e495fbb9f9cd9e2115046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.500273 9234 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1e3635139e495fbb9f9cd9e2115046 (DEFAULT_TABLE table=test-table [id=907dc575fbab48bb86f8272d96fda2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:44.500384 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1e3635139e495fbb9f9cd9e2115046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.501590 9326 tablet_bootstrap.cc:492] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: Bootstrap starting.
I20260926 06:57:44.501711 9094 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1e3635139e495fbb9f9cd9e2115046 (DEFAULT_TABLE table=test-table [id=907dc575fbab48bb86f8272d96fda2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:44.501811 9094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1e3635139e495fbb9f9cd9e2115046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.502053 9326 tablet_bootstrap.cc:654] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.502193 9350 ts_tablet_manager.cc:1905] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:44.503190 9325 tablet_bootstrap.cc:492] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: Bootstrap starting.
I20260926 06:57:44.503592 9325 tablet_bootstrap.cc:654] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.503747 9349 tablet_replica.cc:330] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: stopping tablet replica
I20260926 06:57:44.503878 9349 raft_consensus.cc:2229] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:44.503906 9329 tablet_bootstrap.cc:492] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: Bootstrap starting.
I20260926 06:57:44.504004 9349 raft_consensus.cc:2258] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:44.504181 9350 ts_tablet_manager.cc:1918] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:44.504264 9350 log.cc:1196] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/d943db3aacf144cfa64d1b3316b4fcaa
I20260926 06:57:44.504302 9329 tablet_bootstrap.cc:654] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.504521 9350 ts_tablet_manager.cc:1939] T d943db3aacf144cfa64d1b3316b4fcaa P 9d308dfc112d4943940a9c226e275123: Deleting consensus metadata
I20260926 06:57:44.504819 9325 tablet_bootstrap.cc:492] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: No bootstrap required, opened a new log
I20260926 06:57:44.504869 9325 ts_tablet_manager.cc:1397] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:57:44.505016 9325 raft_consensus.cc:348] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.505098 9325 raft_consensus.cc:374] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.505120 9325 raft_consensus.cc:729] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d308dfc112d4943940a9c226e275123, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.505155 9325 consensus_queue.cc:260] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.505226 9349 ts_tablet_manager.cc:1905] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:44.505241 9325 ts_tablet_manager.cc:1428] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.505633 8804 catalog_manager.cc:4841] TS 9d308dfc112d4943940a9c226e275123 (127.1.74.67:39163): tablet d943db3aacf144cfa64d1b3316b4fcaa (table test-table [id=9b104da7241d41b29b571e6d7d4b3b9a]) successfully deleted
I20260926 06:57:44.506137 9348 ts_tablet_manager.cc:1905] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:44.506994 9349 ts_tablet_manager.cc:1918] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:44.507071 9349 log.cc:1196] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/d943db3aacf144cfa64d1b3316b4fcaa
I20260926 06:57:44.507359 9349 ts_tablet_manager.cc:1939] T d943db3aacf144cfa64d1b3316b4fcaa P b953454746c6422699d2bdff3fa11138: Deleting consensus metadata
I20260926 06:57:44.507658 9329 tablet_bootstrap.cc:492] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: No bootstrap required, opened a new log
I20260926 06:57:44.507665 9348 ts_tablet_manager.cc:1918] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:44.507735 9329 ts_tablet_manager.cc:1397] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260926 06:57:44.507761 9348 log.cc:1196] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/d943db3aacf144cfa64d1b3316b4fcaa
I20260926 06:57:44.507738 8803 catalog_manager.cc:4841] TS b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627): tablet d943db3aacf144cfa64d1b3316b4fcaa (table test-table [id=9b104da7241d41b29b571e6d7d4b3b9a]) successfully deleted
I20260926 06:57:44.507889 9329 raft_consensus.cc:348] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.507974 9329 raft_consensus.cc:374] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.507984 9348 ts_tablet_manager.cc:1939] T d943db3aacf144cfa64d1b3316b4fcaa P f6e48a24c52c47cd9272e242365fe560: Deleting consensus metadata
I20260926 06:57:44.508014 9329 raft_consensus.cc:729] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e48a24c52c47cd9272e242365fe560, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.508068 9329 consensus_queue.cc:260] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.508152 9329 ts_tablet_manager.cc:1428] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.508406 8803 catalog_manager.cc:4841] TS f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597): tablet d943db3aacf144cfa64d1b3316b4fcaa (table test-table [id=9b104da7241d41b29b571e6d7d4b3b9a]) successfully deleted
I20260926 06:57:44.512610 9326 tablet_bootstrap.cc:492] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: No bootstrap required, opened a new log
I20260926 06:57:44.512653 9326 ts_tablet_manager.cc:1397] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260926 06:57:44.512804 9326 raft_consensus.cc:348] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.512861 9326 raft_consensus.cc:374] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.512873 9326 raft_consensus.cc:729] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b953454746c6422699d2bdff3fa11138, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.512925 9326 consensus_queue.cc:260] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.513015 9326 ts_tablet_manager.cc:1428] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.520895 9330 raft_consensus.cc:482] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:44.520972 9330 raft_consensus.cc:504] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.521282 9330 leader_election.cc:288] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627), f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597)
I20260926 06:57:44.524374 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.521473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:43627 (local address 127.1.74.67:40925)
0926 06:57:44.521513 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.521525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.521581 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.521772 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.521772 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.521774 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:44.521839 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.521841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.522552 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.522553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.523424 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.523426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.524102 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.524103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.524176 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.524179 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.524233 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.524277 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 06:57:44.524457 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.521506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:40925 (local address 127.1.74.65:43627)
0926 06:57:44.521556 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.521558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.521566 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:44.521668 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.521669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.521680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.521728 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:44.521914 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.522469 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.523518 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.524016 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.524101 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.524111 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.524348 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.524389 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:57:44.524813 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1e3635139e495fbb9f9cd9e2115046" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b953454746c6422699d2bdff3fa11138" is_pre_election: true
I20260926 06:57:44.524968 8974 raft_consensus.cc:2454] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 0.
I20260926 06:57:44.525156 9187 leader_election.cc:302] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d308dfc112d4943940a9c226e275123, b953454746c6422699d2bdff3fa11138; no voters:
I20260926 06:57:44.525285 9330 raft_consensus.cc:2781] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:44.525348 9330 raft_consensus.cc:482] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:44.525357 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.521746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:33705 (local address 127.1.74.66:38597)
0926 06:57:44.521794 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.521796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.521805 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:44.521831 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.521832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.521841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.521902 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:57:44.522143 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.522850 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.524302 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.524875 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.524951 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.524960 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.525223 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.525272 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:57:44.525380 9330 raft_consensus.cc:3037] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:44.525512 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.521579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:38597 (local address 127.1.74.67:33705)
0926 06:57:44.521670 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.521680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.521723 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.521952 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.521953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.521955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:57:44.522040 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.522041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.522957 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.522958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.524212 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.524215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.524964 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.524965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.525117 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.525122 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.525210 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.525249 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
I20260926 06:57:44.525733 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1e3635139e495fbb9f9cd9e2115046" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560" is_pre_election: true
I20260926 06:57:44.525810 9114 raft_consensus.cc:2454] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 0.
I20260926 06:57:44.526262 9330 raft_consensus.cc:504] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.526414 9330 leader_election.cc:288] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 election: Requested vote from peers b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627), f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597)
I20260926 06:57:44.526587 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1e3635139e495fbb9f9cd9e2115046" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b953454746c6422699d2bdff3fa11138"
I20260926 06:57:44.526648 8974 raft_consensus.cc:3037] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:44.526607 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1e3635139e495fbb9f9cd9e2115046" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560"
I20260926 06:57:44.526684 9114 raft_consensus.cc:3037] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:44.527427 8974 raft_consensus.cc:2454] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 1.
I20260926 06:57:44.527511 9114 raft_consensus.cc:2454] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 1.
I20260926 06:57:44.527590 9187 leader_election.cc:302] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d308dfc112d4943940a9c226e275123, b953454746c6422699d2bdff3fa11138; no voters:
I20260926 06:57:44.527660 9330 raft_consensus.cc:2781] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:44.527796 9330 raft_consensus.cc:686] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 1 LEADER]: Becoming Leader. State: Replica: 9d308dfc112d4943940a9c226e275123, State: Running, Role: LEADER
I20260926 06:57:44.527864 9330 consensus_queue.cc:237] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.528524 8816 catalog_manager.cc:5495] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d308dfc112d4943940a9c226e275123 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } health_report { overall_health: HEALTHY } } }
W20260926 06:57:44.540987 9021 tablet.cc:2369] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405863, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.126@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:44.563810 9362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.555941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:40563 (local address 127.0.0.1:46492)
0926 06:57:44.556086 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.556105 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.556163 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.556311 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.556312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.556598 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:44.556643 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.556645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.557574 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.557575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.558071 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.558072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.558526 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.558527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.558564 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.559148 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:44.559152 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:44.560852 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:44.562630 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.562636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.562638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.562851 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.563082 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:44.563083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:44.563083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:44.563136 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:44.563470 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:44.563477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:44.563523 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.563592 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.563688 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260926 06:57:44.563884 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.556007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.1.74.126:40563)
0926 06:57:44.556066 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.556068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.556168 (+ 100us) server_negotiation.cc:407] Connection header received
0926 06:57:44.556204 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.556204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.556217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.556299 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.556720 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.557485 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.558153 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.558443 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.558515 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.560944 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:44.560945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:44.560946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:44.560954 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:44.562546 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.562934 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.562935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.562936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.563003 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.563199 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.563200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.563200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.563368 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:44.563445 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.563644 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.563764 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":34}
I20260926 06:57:44.566030 8816 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:46492:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 06:57:44.566246 8816 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:44.567893 8954 tablet_service.cc:1467] Processing CreateTablet for tablet 98c3d40c04934f3e8ff4013c3f188abe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fbfd09accd27403eb3a665ca9867fafe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:57:44.567898 9094 tablet_service.cc:1467] Processing CreateTablet for tablet 98c3d40c04934f3e8ff4013c3f188abe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fbfd09accd27403eb3a665ca9867fafe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:57:44.567912 9234 tablet_service.cc:1467] Processing CreateTablet for tablet 98c3d40c04934f3e8ff4013c3f188abe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fbfd09accd27403eb3a665ca9867fafe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:57:44.568042 9094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c3d40c04934f3e8ff4013c3f188abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.568042 8954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c3d40c04934f3e8ff4013c3f188abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.568068 9234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c3d40c04934f3e8ff4013c3f188abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.569517 9326 tablet_bootstrap.cc:492] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: Bootstrap starting.
I20260926 06:57:44.569525 9329 tablet_bootstrap.cc:492] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: Bootstrap starting.
I20260926 06:57:44.569766 9325 tablet_bootstrap.cc:492] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: Bootstrap starting.
I20260926 06:57:44.569835 9326 tablet_bootstrap.cc:654] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.569895 9329 tablet_bootstrap.cc:654] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.570178 9325 tablet_bootstrap.cc:654] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.570523 9329 tablet_bootstrap.cc:492] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: No bootstrap required, opened a new log
I20260926 06:57:44.570571 9329 ts_tablet_manager.cc:1397] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:44.570734 9329 raft_consensus.cc:348] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.570815 9329 raft_consensus.cc:374] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.570842 9329 raft_consensus.cc:729] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e48a24c52c47cd9272e242365fe560, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.570889 9329 consensus_queue.cc:260] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.570945 9326 tablet_bootstrap.cc:492] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: No bootstrap required, opened a new log
I20260926 06:57:44.570981 9326 ts_tablet_manager.cc:1397] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:44.570981 9329 ts_tablet_manager.cc:1428] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:44.571126 9325 tablet_bootstrap.cc:492] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: No bootstrap required, opened a new log
I20260926 06:57:44.571108 9326 raft_consensus.cc:348] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.571115 9333 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.571171 9326 raft_consensus.cc:374] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.571180 9333 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:44.571201 9326 raft_consensus.cc:729] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b953454746c6422699d2bdff3fa11138, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.571172 9325 ts_tablet_manager.cc:1397] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:44.571241 9326 consensus_queue.cc:260] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.571359 9326 ts_tablet_manager.cc:1428] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:44.571378 9325 raft_consensus.cc:348] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.571448 9325 raft_consensus.cc:374] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.571471 9325 raft_consensus.cc:729] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d308dfc112d4943940a9c226e275123, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.571520 9325 consensus_queue.cc:260] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.571576 9345 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.571617 9325 ts_tablet_manager.cc:1428] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:44.571651 9345 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:44.572119 9330 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.572245 9330 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 06:57:44.598176 9301 tablet.cc:2369] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:57:44.671101 9330 raft_consensus.cc:482] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:44.671217 9330 raft_consensus.cc:504] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.671439 9330 leader_election.cc:288] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627), f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597)
I20260926 06:57:44.671785 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c3d40c04934f3e8ff4013c3f188abe" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560" is_pre_election: true
I20260926 06:57:44.671904 9114 raft_consensus.cc:2454] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 0.
I20260926 06:57:44.671840 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c3d40c04934f3e8ff4013c3f188abe" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b953454746c6422699d2bdff3fa11138" is_pre_election: true
I20260926 06:57:44.672010 8974 raft_consensus.cc:2454] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 0.
I20260926 06:57:44.672076 9187 leader_election.cc:302] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d308dfc112d4943940a9c226e275123, f6e48a24c52c47cd9272e242365fe560; no voters:
I20260926 06:57:44.672176 9330 raft_consensus.cc:2781] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:44.672235 9330 raft_consensus.cc:482] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:44.672252 9330 raft_consensus.cc:3037] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:44.673074 9330 raft_consensus.cc:504] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.673213 9330 leader_election.cc:288] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 election: Requested vote from peers b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627), f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597)
I20260926 06:57:44.673408 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c3d40c04934f3e8ff4013c3f188abe" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b953454746c6422699d2bdff3fa11138"
I20260926 06:57:44.673439 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c3d40c04934f3e8ff4013c3f188abe" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560"
I20260926 06:57:44.673506 8974 raft_consensus.cc:3037] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:44.673513 9114 raft_consensus.cc:3037] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:44.674118 8974 raft_consensus.cc:2454] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 1.
I20260926 06:57:44.674190 9114 raft_consensus.cc:2454] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 1.
I20260926 06:57:44.674345 9187 leader_election.cc:302] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d308dfc112d4943940a9c226e275123, b953454746c6422699d2bdff3fa11138; no voters:
I20260926 06:57:44.674461 9330 raft_consensus.cc:2781] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:44.674548 9330 raft_consensus.cc:686] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [term 1 LEADER]: Becoming Leader. State: Replica: 9d308dfc112d4943940a9c226e275123, State: Running, Role: LEADER
I20260926 06:57:44.674608 9330 consensus_queue.cc:237] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.674944 9355 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: TxnStatusTablet state changed. Reason: New leader 9d308dfc112d4943940a9c226e275123. Latest consensus state: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.675052 9355 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:57:44.675235 9367 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 06:57:44.675411 9367 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 06:57:44.675452 8817 catalog_manager.cc:5495] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d308dfc112d4943940a9c226e275123 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } health_report { overall_health: HEALTHY } } }
I20260926 06:57:44.688855 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.686423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.1.74.67:39163)
0926 06:57:44.686469 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.686471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.686480 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:44.686555 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.686556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.686573 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.686657 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:44.686765 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.687303 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.688052 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.688261 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.688348 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.688533 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:44.688635 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:44.688715 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.688763 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.688786 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:57:44.688892 9362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.686363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:39163 (local address 127.0.0.1:36526)
0926 06:57:44.686418 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.686430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.686476 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.686663 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.686664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.686665 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:44.686713 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.686715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.687380 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.687381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.687961 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.687963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.688353 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.688355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.688420 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.688440 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:44.688701 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:44.688710 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.688759 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.688816 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 06:57:44.689996 9114 raft_consensus.cc:1261] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Refusing update from remote peer 9d308dfc112d4943940a9c226e275123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:44.690073 8974 raft_consensus.cc:1261] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Refusing update from remote peer 9d308dfc112d4943940a9c226e275123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:44.690248 9355 consensus_queue.cc:1035] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:44.690342 9330 consensus_queue.cc:1035] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:44.690981 9333 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: TxnStatusTablet state changed. Reason: New leader 9d308dfc112d4943940a9c226e275123. Latest consensus state: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.691073 9333 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:44.691076 9345 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: TxnStatusTablet state changed. Reason: New leader 9d308dfc112d4943940a9c226e275123. Latest consensus state: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.691161 9345 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:44.691751 9355 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.691826 9355 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:57:44.692049 9345 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.692138 9345 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P b953454746c6422699d2bdff3fa11138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:44.693042 9330 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.693118 9330 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P 9d308dfc112d4943940a9c226e275123: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:57:44.693636 9333 tablet_replica.cc:439] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } } }
I20260926 06:57:44.693725 9333 tablet_replica.cc:442] T 98c3d40c04934f3e8ff4013c3f188abe P f6e48a24c52c47cd9272e242365fe560: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:57:44.696024 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.693185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.1.74.67:39163)
0926 06:57:44.693365 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.693367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.693376 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:44.693396 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.693396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.693407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.693499 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:44.693850 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.694547 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.695287 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.695573 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.695652 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.695769 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:44.695803 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:44.695863 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.695932 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.695952 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260926 06:57:44.700928 9306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.698592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:39163 (local address 127.1.74.67:59861)
0926 06:57:44.698682 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:44.698694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:44.698741 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:44.698940 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:44.698941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:44.698942 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:57:44.699001 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.699002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.699625 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.699626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.700185 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:44.700186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.700534 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.700534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:44.700596 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.700607 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:57:44.700784 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:57:44.700785 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:57:44.700837 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 06:57:44.700870 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 06:57:44.700928 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.698644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:59861 (local address 127.1.74.67:39163)
0926 06:57:44.698703 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.698705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.698727 (+ 22us) server_negotiation.cc:407] Connection header received
0926 06:57:44.698858 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.698858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.698866 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.698935 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:44.699063 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.699552 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.700279 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.700455 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.700533 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.700693 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:44.700726 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:44.700785 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.700841 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.700854 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:57:44.703485 8974 raft_consensus.cc:1261] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Refusing update from remote peer 9d308dfc112d4943940a9c226e275123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:44.703496 9114 raft_consensus.cc:1261] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Refusing update from remote peer 9d308dfc112d4943940a9c226e275123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:44.703730 9330 consensus_queue.cc:1035] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:44.703857 9374 consensus_queue.cc:1035] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:44.705281 9340 mvcc.cc:205] Tried to move back new op lower bound from 7333502421823934464 to 7333502421106651136. Current Snapshot: MvccSnapshot[applied={T|T < 7333502421823934464}]
I20260926 06:57:44.718304 8815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:44.718428 8815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:44.719051 8815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e: Sending DeleteTablet for 3 replicas of tablet 9e1e3635139e495fbb9f9cd9e2115046
I20260926 06:57:44.719403 8954 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1e3635139e495fbb9f9cd9e2115046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:44 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57674
I20260926 06:57:44.719403 9233 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1e3635139e495fbb9f9cd9e2115046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:44 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:36514
I20260926 06:57:44.719506 9350 tablet_replica.cc:330] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: stopping tablet replica
I20260926 06:57:44.719516 9349 tablet_replica.cc:330] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: stopping tablet replica
I20260926 06:57:44.719560 9349 raft_consensus.cc:2229] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:44.719558 9350 raft_consensus.cc:2229] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:44.719597 9349 raft_consensus.cc:2258] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:44.719646 9350 raft_consensus.cc:2258] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:44.719871 9349 ts_tablet_manager.cc:1905] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:44.719897 9094 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1e3635139e495fbb9f9cd9e2115046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:44 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:41722
I20260926 06:57:44.719998 9348 tablet_replica.cc:330] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: stopping tablet replica
I20260926 06:57:44.720009 9350 ts_tablet_manager.cc:1905] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:44.720057 9348 raft_consensus.cc:2229] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:44.720094 9348 raft_consensus.cc:2258] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:44.721411 9349 ts_tablet_manager.cc:1918] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:44.721509 9349 log.cc:1196] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/9e1e3635139e495fbb9f9cd9e2115046
I20260926 06:57:44.721714 9349 ts_tablet_manager.cc:1939] T 9e1e3635139e495fbb9f9cd9e2115046 P b953454746c6422699d2bdff3fa11138: Deleting consensus metadata
I20260926 06:57:44.721918 9350 ts_tablet_manager.cc:1918] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:44.721990 9350 log.cc:1196] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/9e1e3635139e495fbb9f9cd9e2115046
I20260926 06:57:44.722221 9350 ts_tablet_manager.cc:1939] T 9e1e3635139e495fbb9f9cd9e2115046 P 9d308dfc112d4943940a9c226e275123: Deleting consensus metadata
I20260926 06:57:44.722424 8803 catalog_manager.cc:4841] TS b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627): tablet 9e1e3635139e495fbb9f9cd9e2115046 (table test-table [id=907dc575fbab48bb86f8272d96fda2e5]) successfully deleted
I20260926 06:57:44.722591 9348 ts_tablet_manager.cc:1905] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:44.722642 8804 catalog_manager.cc:4841] TS 9d308dfc112d4943940a9c226e275123 (127.1.74.67:39163): tablet 9e1e3635139e495fbb9f9cd9e2115046 (table test-table [id=907dc575fbab48bb86f8272d96fda2e5]) successfully deleted
I20260926 06:57:44.723839 9348 ts_tablet_manager.cc:1918] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:44.723899 9348 log.cc:1196] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/9e1e3635139e495fbb9f9cd9e2115046
I20260926 06:57:44.724149 9348 ts_tablet_manager.cc:1939] T 9e1e3635139e495fbb9f9cd9e2115046 P f6e48a24c52c47cd9272e242365fe560: Deleting consensus metadata
I20260926 06:57:44.724452 8803 catalog_manager.cc:4841] TS f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597): tablet 9e1e3635139e495fbb9f9cd9e2115046 (table test-table [id=907dc575fbab48bb86f8272d96fda2e5]) successfully deleted
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:57:44.745565 8866 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:44.739979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.1.74.126:40563)
0926 06:57:44.740052 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.740054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.740163 (+ 109us) server_negotiation.cc:407] Connection header received
0926 06:57:44.740245 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.740246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.740256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.740312 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.740870 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.741360 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.742173 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.742394 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.742480 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.745498 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46504: BlockingRecv error: recv got EOF from 127.0.0.1:46504 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 06:57:44 dist-test-slave-sfvt krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM
I20260926 06:57:44.753631 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:44.746412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.1.74.126:40563)
0926 06:57:44.746453 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:44.746455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:44.746575 (+ 120us) server_negotiation.cc:407] Connection header received
0926 06:57:44.746639 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:44.746640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:44.746651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:44.746722 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:44.747188 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.747712 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.748437 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:44.748641 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:44.748726 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:44.751107 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:44.751109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:44.751109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:44.751117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:44.752531 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.752875 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.752877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.752877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.752936 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:44.753128 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:44.753129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:44.753129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:44.753241 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:44.753352 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:44.753452 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 06:57:44.753528 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:57:44.755113 8815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:44.755334 8815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:44.756760 8954 tablet_service.cc:1467] Processing CreateTablet for tablet a6c4d8a7c0fa4ee297a6a04c00c346f4 (DEFAULT_TABLE table=test-table [id=3275c9de16894001a90828483050c83b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:44.756783 9094 tablet_service.cc:1467] Processing CreateTablet for tablet a6c4d8a7c0fa4ee297a6a04c00c346f4 (DEFAULT_TABLE table=test-table [id=3275c9de16894001a90828483050c83b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:44.756917 8954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c4d8a7c0fa4ee297a6a04c00c346f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.756922 9094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c4d8a7c0fa4ee297a6a04c00c346f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.758106 9326 tablet_bootstrap.cc:492] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: Bootstrap starting.
I20260926 06:57:44.758347 9233 tablet_service.cc:1467] Processing CreateTablet for tablet a6c4d8a7c0fa4ee297a6a04c00c346f4 (DEFAULT_TABLE table=test-table [id=3275c9de16894001a90828483050c83b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:44.758456 9326 tablet_bootstrap.cc:654] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.758469 9233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c4d8a7c0fa4ee297a6a04c00c346f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:44.759382 9329 tablet_bootstrap.cc:492] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: Bootstrap starting.
I20260926 06:57:44.759724 9329 tablet_bootstrap.cc:654] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.759894 9325 tablet_bootstrap.cc:492] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: Bootstrap starting.
I20260926 06:57:44.760174 9325 tablet_bootstrap.cc:654] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:44.760736 9326 tablet_bootstrap.cc:492] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: No bootstrap required, opened a new log
I20260926 06:57:44.760790 9326 ts_tablet_manager.cc:1397] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 06:57:44.760897 9325 tablet_bootstrap.cc:492] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: No bootstrap required, opened a new log
I20260926 06:57:44.760955 9325 ts_tablet_manager.cc:1397] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:44.760953 9326 raft_consensus.cc:348] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.761036 9326 raft_consensus.cc:374] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.761063 9326 raft_consensus.cc:729] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b953454746c6422699d2bdff3fa11138, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.761107 9326 consensus_queue.cc:260] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.761111 9325 raft_consensus.cc:348] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.761189 9325 raft_consensus.cc:374] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.761194 9326 ts_tablet_manager.cc:1428] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:44.761214 9325 raft_consensus.cc:729] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d308dfc112d4943940a9c226e275123, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.761281 9325 consensus_queue.cc:260] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.761384 9325 ts_tablet_manager.cc:1428] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:44.761865 9329 tablet_bootstrap.cc:492] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: No bootstrap required, opened a new log
I20260926 06:57:44.761938 9329 ts_tablet_manager.cc:1397] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 06:57:44.762087 9329 raft_consensus.cc:348] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.762146 9329 raft_consensus.cc:374] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:44.762159 9329 raft_consensus.cc:729] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e48a24c52c47cd9272e242365fe560, State: Initialized, Role: FOLLOWER
I20260926 06:57:44.762212 9329 consensus_queue.cc:260] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:44.762313 9329 ts_tablet_manager.cc:1428] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:45.003532 9374 raft_consensus.cc:482] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:45.003634 9374 raft_consensus.cc:504] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.003826 9374 leader_election.cc:288] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627), f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597)
I20260926 06:57:45.004156 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c4d8a7c0fa4ee297a6a04c00c346f4" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560" is_pre_election: true
I20260926 06:57:45.004144 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c4d8a7c0fa4ee297a6a04c00c346f4" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b953454746c6422699d2bdff3fa11138" is_pre_election: true
I20260926 06:57:45.004251 9114 raft_consensus.cc:2454] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 0.
I20260926 06:57:45.004251 8974 raft_consensus.cc:2454] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 0.
I20260926 06:57:45.004431 9187 leader_election.cc:302] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d308dfc112d4943940a9c226e275123, b953454746c6422699d2bdff3fa11138; no voters:
I20260926 06:57:45.004531 9374 raft_consensus.cc:2781] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:45.004576 9374 raft_consensus.cc:482] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:45.004601 9374 raft_consensus.cc:3037] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:45.005261 9374 raft_consensus.cc:504] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.005380 9374 leader_election.cc:288] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 election: Requested vote from peers b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627), f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597)
I20260926 06:57:45.005561 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c4d8a7c0fa4ee297a6a04c00c346f4" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b953454746c6422699d2bdff3fa11138"
I20260926 06:57:45.005656 8974 raft_consensus.cc:3037] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:45.005607 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c4d8a7c0fa4ee297a6a04c00c346f4" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560"
I20260926 06:57:45.005700 9114 raft_consensus.cc:3037] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:45.006265 9114 raft_consensus.cc:2454] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 1.
I20260926 06:57:45.006385 8974 raft_consensus.cc:2454] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 1.
I20260926 06:57:45.006444 9187 leader_election.cc:302] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d308dfc112d4943940a9c226e275123, f6e48a24c52c47cd9272e242365fe560; no voters:
I20260926 06:57:45.006557 9374 raft_consensus.cc:2781] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:45.006608 9374 raft_consensus.cc:686] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 1 LEADER]: Becoming Leader. State: Replica: 9d308dfc112d4943940a9c226e275123, State: Running, Role: LEADER
I20260926 06:57:45.006662 9374 consensus_queue.cc:237] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.007236 8816 catalog_manager.cc:5495] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d308dfc112d4943940a9c226e275123 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } health_report { overall_health: HEALTHY } } }
W20260926 06:57:45.042858 9015 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.1.74.65:43627
I20260926 06:57:45.044229 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:45.041320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.1.74.67:39163)
0926 06:57:45.041381 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:45.041383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:45.041393 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:45.041419 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:45.041420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:45.041430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:45.041493 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:57:45.041656 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:45.042400 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:45.043380 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:45.043679 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:45.043771 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:45.043949 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:57:45.043986 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:57:45.044043 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:45.044124 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 06:57:45.044149 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:57:45.045010 9114 raft_consensus.cc:1261] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Refusing update from remote peer 9d308dfc112d4943940a9c226e275123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:45.045042 8974 raft_consensus.cc:1261] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Refusing update from remote peer 9d308dfc112d4943940a9c226e275123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:45.045240 9374 consensus_queue.cc:1035] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:45.045392 9330 consensus_queue.cc:1035] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:45.046638 9338 mvcc.cc:205] Tried to move back new op lower bound from 7333502423222624256 to 7333502423067578368. Current Snapshot: MvccSnapshot[applied={T|T < 7333502423222624256}]
I20260926 06:57:45.047871 8815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:45.047974 8815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:45.048686 8815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e: Sending DeleteTablet for 3 replicas of tablet a6c4d8a7c0fa4ee297a6a04c00c346f4
I20260926 06:57:45.048945 8954 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c4d8a7c0fa4ee297a6a04c00c346f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:45 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57674
I20260926 06:57:45.048970 9094 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c4d8a7c0fa4ee297a6a04c00c346f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:45 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:41722
I20260926 06:57:45.049016 9233 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c4d8a7c0fa4ee297a6a04c00c346f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:45 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:36514
I20260926 06:57:45.049037 9349 tablet_replica.cc:330] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: stopping tablet replica
I20260926 06:57:45.049088 9349 raft_consensus.cc:2229] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:45.049105 9348 tablet_replica.cc:330] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: stopping tablet replica
I20260926 06:57:45.049103 9350 tablet_replica.cc:330] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: stopping tablet replica
I20260926 06:57:45.049129 9349 raft_consensus.cc:2258] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:45.049163 9348 raft_consensus.cc:2229] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:45.049171 9350 raft_consensus.cc:2229] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:45.049212 9348 raft_consensus.cc:2258] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:45.049233 9350 raft_consensus.cc:2258] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:45.049510 9349 ts_tablet_manager.cc:1905] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:45.049577 9350 ts_tablet_manager.cc:1905] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:45.049646 9348 ts_tablet_manager.cc:1905] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:45.051178 9350 ts_tablet_manager.cc:1918] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:45.051208 9348 ts_tablet_manager.cc:1918] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:45.051231 9350 log.cc:1196] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/a6c4d8a7c0fa4ee297a6a04c00c346f4
I20260926 06:57:45.051306 9348 log.cc:1196] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/a6c4d8a7c0fa4ee297a6a04c00c346f4
I20260926 06:57:45.051236 8815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:57:45.051415 8815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:57:45.051506 9350 ts_tablet_manager.cc:1939] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P 9d308dfc112d4943940a9c226e275123: Deleting consensus metadata
I20260926 06:57:45.051517 9348 ts_tablet_manager.cc:1939] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P f6e48a24c52c47cd9272e242365fe560: Deleting consensus metadata
I20260926 06:57:45.051813 9349 ts_tablet_manager.cc:1918] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:57:45.051820 8804 catalog_manager.cc:4841] TS 9d308dfc112d4943940a9c226e275123 (127.1.74.67:39163): tablet a6c4d8a7c0fa4ee297a6a04c00c346f4 (table test-table [id=3275c9de16894001a90828483050c83b]) successfully deleted
I20260926 06:57:45.051858 9349 log.cc:1196] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/a6c4d8a7c0fa4ee297a6a04c00c346f4
I20260926 06:57:45.051941 8803 catalog_manager.cc:4841] TS f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597): tablet a6c4d8a7c0fa4ee297a6a04c00c346f4 (table test-table [id=3275c9de16894001a90828483050c83b]) successfully deleted
I20260926 06:57:45.052083 9349 ts_tablet_manager.cc:1939] T a6c4d8a7c0fa4ee297a6a04c00c346f4 P b953454746c6422699d2bdff3fa11138: Deleting consensus metadata
I20260926 06:57:45.052388 8803 catalog_manager.cc:4841] TS b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627): tablet a6c4d8a7c0fa4ee297a6a04c00c346f4 (table test-table [id=3275c9de16894001a90828483050c83b]) successfully deleted
I20260926 06:57:45.052618 8954 tablet_service.cc:1467] Processing CreateTablet for tablet f38bf44ebb0e4ef781e2e21634e29f2a (DEFAULT_TABLE table=test-table [id=3e0a68ba29604fc9a1e0265fd799c250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:45.052629 9233 tablet_service.cc:1467] Processing CreateTablet for tablet f38bf44ebb0e4ef781e2e21634e29f2a (DEFAULT_TABLE table=test-table [id=3e0a68ba29604fc9a1e0265fd799c250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:45.052632 9094 tablet_service.cc:1467] Processing CreateTablet for tablet f38bf44ebb0e4ef781e2e21634e29f2a (DEFAULT_TABLE table=test-table [id=3e0a68ba29604fc9a1e0265fd799c250]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:57:45.052721 8954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38bf44ebb0e4ef781e2e21634e29f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:45.052721 9233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38bf44ebb0e4ef781e2e21634e29f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:45.052729 9094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38bf44ebb0e4ef781e2e21634e29f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:45.053879 9325 tablet_bootstrap.cc:492] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: Bootstrap starting.
I20260926 06:57:45.053882 9329 tablet_bootstrap.cc:492] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: Bootstrap starting.
I20260926 06:57:45.054188 9325 tablet_bootstrap.cc:654] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:45.054345 9329 tablet_bootstrap.cc:654] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:45.054461 9326 tablet_bootstrap.cc:492] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138: Bootstrap starting.
I20260926 06:57:45.054795 9326 tablet_bootstrap.cc:654] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:45.054958 9329 tablet_bootstrap.cc:492] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: No bootstrap required, opened a new log
I20260926 06:57:45.055025 9329 ts_tablet_manager.cc:1397] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:45.055171 9325 tablet_bootstrap.cc:492] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: No bootstrap required, opened a new log
I20260926 06:57:45.055181 9329 raft_consensus.cc:348] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.055236 9325 ts_tablet_manager.cc:1397] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:45.055246 9329 raft_consensus.cc:374] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:45.055310 9329 raft_consensus.cc:729] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e48a24c52c47cd9272e242365fe560, State: Initialized, Role: FOLLOWER
I20260926 06:57:45.055364 9326 tablet_bootstrap.cc:492] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138: No bootstrap required, opened a new log
I20260926 06:57:45.055356 9329 consensus_queue.cc:260] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.055356 9325 raft_consensus.cc:348] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.055400 9326 ts_tablet_manager.cc:1397] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:45.055424 9325 raft_consensus.cc:374] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:45.055449 9325 raft_consensus.cc:729] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d308dfc112d4943940a9c226e275123, State: Initialized, Role: FOLLOWER
I20260926 06:57:45.055447 9329 ts_tablet_manager.cc:1428] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:45.055493 9325 consensus_queue.cc:260] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.055493 9326 raft_consensus.cc:348] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.055564 9326 raft_consensus.cc:374] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:45.055584 9326 raft_consensus.cc:729] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b953454746c6422699d2bdff3fa11138, State: Initialized, Role: FOLLOWER
I20260926 06:57:45.055584 9325 ts_tablet_manager.cc:1428] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:45.055631 9326 consensus_queue.cc:260] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.055712 9326 ts_tablet_manager.cc:1428] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:45.240592 9330 raft_consensus.cc:482] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:57:45.240684 9330 raft_consensus.cc:504] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.240830 9330 leader_election.cc:288] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627), f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597)
I20260926 06:57:45.241168 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38bf44ebb0e4ef781e2e21634e29f2a" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b953454746c6422699d2bdff3fa11138" is_pre_election: true
I20260926 06:57:45.241255 8974 raft_consensus.cc:2454] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 0.
I20260926 06:57:45.241189 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38bf44ebb0e4ef781e2e21634e29f2a" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560" is_pre_election: true
I20260926 06:57:45.241309 9114 raft_consensus.cc:2454] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 0.
I20260926 06:57:45.241449 9187 leader_election.cc:302] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d308dfc112d4943940a9c226e275123, b953454746c6422699d2bdff3fa11138; no voters:
I20260926 06:57:45.241561 9330 raft_consensus.cc:2781] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:57:45.241618 9330 raft_consensus.cc:482] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:57:45.241649 9330 raft_consensus.cc:3037] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:45.242422 9330 raft_consensus.cc:504] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.242587 9330 leader_election.cc:288] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 election: Requested vote from peers b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627), f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597)
I20260926 06:57:45.242743 8974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38bf44ebb0e4ef781e2e21634e29f2a" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b953454746c6422699d2bdff3fa11138"
I20260926 06:57:45.242782 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38bf44ebb0e4ef781e2e21634e29f2a" candidate_uuid: "9d308dfc112d4943940a9c226e275123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e48a24c52c47cd9272e242365fe560"
I20260926 06:57:45.242847 9114 raft_consensus.cc:3037] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:45.242846 8974 raft_consensus.cc:3037] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:45.243578 9114 raft_consensus.cc:2454] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 1.
I20260926 06:57:45.243674 8974 raft_consensus.cc:2454] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d308dfc112d4943940a9c226e275123 in term 1.
I20260926 06:57:45.243772 9187 leader_election.cc:302] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d308dfc112d4943940a9c226e275123, f6e48a24c52c47cd9272e242365fe560; no voters:
I20260926 06:57:45.243888 9330 raft_consensus.cc:2781] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:45.243948 9330 raft_consensus.cc:686] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 1 LEADER]: Becoming Leader. State: Replica: 9d308dfc112d4943940a9c226e275123, State: Running, Role: LEADER
I20260926 06:57:45.244002 9330 consensus_queue.cc:237] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } }
I20260926 06:57:45.244572 8817 catalog_manager.cc:5495] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d308dfc112d4943940a9c226e275123 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "9d308dfc112d4943940a9c226e275123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d308dfc112d4943940a9c226e275123" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39163 } health_report { overall_health: HEALTHY } } }
I20260926 06:57:45.284868 8974 raft_consensus.cc:1261] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Refusing update from remote peer 9d308dfc112d4943940a9c226e275123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:45.284924 9114 raft_consensus.cc:1261] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Refusing update from remote peer 9d308dfc112d4943940a9c226e275123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:57:45.285187 9355 consensus_queue.cc:1035] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b953454746c6422699d2bdff3fa11138" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 43627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:45.285305 9330 consensus_queue.cc:1035] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e48a24c52c47cd9272e242365fe560" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 38597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:57:45.294476 8815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:45.294569 8815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:57:45.295146 8815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fe394fb1cc443cd87694a36ac43957e: Sending DeleteTablet for 3 replicas of tablet f38bf44ebb0e4ef781e2e21634e29f2a
I20260926 06:57:45.295394 9233 tablet_service.cc:1514] Processing DeleteTablet for tablet f38bf44ebb0e4ef781e2e21634e29f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:45 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:36514
I20260926 06:57:45.295394 8954 tablet_service.cc:1514] Processing DeleteTablet for tablet f38bf44ebb0e4ef781e2e21634e29f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:45 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57674
I20260926 06:57:45.295415 9094 tablet_service.cc:1514] Processing DeleteTablet for tablet f38bf44ebb0e4ef781e2e21634e29f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:57:45 UTC) from {username='oryx', principal='oryx/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:41722
I20260926 06:57:45.295483 9349 tablet_replica.cc:330] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138: stopping tablet replica
I20260926 06:57:45.295506 9350 tablet_replica.cc:330] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: stopping tablet replica
I20260926 06:57:45.295509 9348 tablet_replica.cc:330] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: stopping tablet replica
I20260926 06:57:45.295536 9349 raft_consensus.cc:2229] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:45.295560 9349 raft_consensus.cc:2258] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:45.295562 9348 raft_consensus.cc:2229] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:57:45.295562 9350 raft_consensus.cc:2229] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:57:45.295594 9348 raft_consensus.cc:2258] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:45.295612 9350 raft_consensus.cc:2258] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:57:45.295857 9349 ts_tablet_manager.cc:1905] T f38bf44ebb0e4ef781e2e21634e29f2a P b953454746c6422699d2bdff3fa11138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:45.295967 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 8889
I20260926 06:57:45.295994 9350 ts_tablet_manager.cc:1905] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:45.297771 9350 ts_tablet_manager.cc:1918] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:45.297811 9350 log.cc:1196] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/f38bf44ebb0e4ef781e2e21634e29f2a
I20260926 06:57:45.297988 9350 ts_tablet_manager.cc:1939] T f38bf44ebb0e4ef781e2e21634e29f2a P 9d308dfc112d4943940a9c226e275123: Deleting consensus metadata
I20260926 06:57:45.298357 8804 catalog_manager.cc:4841] TS 9d308dfc112d4943940a9c226e275123 (127.1.74.67:39163): tablet f38bf44ebb0e4ef781e2e21634e29f2a (table test-table [id=3e0a68ba29604fc9a1e0265fd799c250]) successfully deleted
I20260926 06:57:45.299955 9348 ts_tablet_manager.cc:1905] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:57:45.301751 9348 ts_tablet_manager.cc:1918] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:57:45.301815 9348 log.cc:1196] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/f38bf44ebb0e4ef781e2e21634e29f2a
I20260926 06:57:45.302018 9348 ts_tablet_manager.cc:1939] T f38bf44ebb0e4ef781e2e21634e29f2a P f6e48a24c52c47cd9272e242365fe560: Deleting consensus metadata
I20260926 06:57:45.302541 8803 catalog_manager.cc:4841] TS f6e48a24c52c47cd9272e242365fe560 (127.1.74.66:38597): tablet f38bf44ebb0e4ef781e2e21634e29f2a (table test-table [id=3e0a68ba29604fc9a1e0265fd799c250]) successfully deleted
W20260926 06:57:45.303643 8803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:43627 (error 108)
I20260926 06:57:45.303925 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 9029
I20260926 06:57:45.304605 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:45.303975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:43627 (local address 127.0.0.1:57696)
0926 06:57:45.304505 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:45.304556 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":38,"threads_started":1}
W20260926 06:57:45.304852 8803 catalog_manager.cc:4584] TS b953454746c6422699d2bdff3fa11138 (127.1.74.65:43627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f38bf44ebb0e4ef781e2e21634e29f2a: Network error: Client connection negotiation failed: client connection to 127.1.74.65:43627: connect: Connection refused (error 111)
I20260926 06:57:45.309006 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 9169
I20260926 06:57:45.314368 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 8785
2026-09-26T06:57:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2730 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:45 dist-test-slave-sfvt krb5kdc[9409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:45 dist-test-slave-sfvt krb5kdc[9409](info): set up 2 sockets
Sep 26 06:57:45 dist-test-slave-sfvt krb5kdc[9409](info): commencing operation
krb5kdc: starting...
W20260926 06:57:46.355886 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:46 dist-test-slave-sfvt krb5kdc[9409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:46.475068 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:41847
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:41847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:46.556602 9422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:46.556810 9422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:46.556833 9422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:46.558125 9422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:46.558172 9422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:46.558187 9422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:46.558199 9422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:46.558211 9422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:46.559790 9422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:41847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:41847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:46.560046 9422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:46.560292 9422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:46.565097 9427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:46.565132 9428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:46.565094 9430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:46.565178 9422 server_base.cc:1034] running on GCE node
I20260926 06:57:46.566566 9422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:46.566865 9422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:46.568060 9422 hybrid_clock.cc:648] HybridClock initialized: now 1790405866568029 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T06:57:46Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1268 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:57:46 dist-test-slave-sfvt krb5kdc[9435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:57:46 dist-test-slave-sfvt krb5kdc[9435](info): set up 2 sockets
Sep 26 06:57:46 dist-test-slave-sfvt krb5kdc[9435](info): commencing operation
krb5kdc: starting...
W20260926 06:57:47.619951 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:57:47 dist-test-slave-sfvt krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:57:47.736264 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:38701
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:38701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:47.814465 9448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:47.814731 9448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:47.814769 9448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:47.815943 9448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:47.816012 9448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:47.816035 9448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:47.816057 9448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:47.816085 9448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:47.817521 9448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:38701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:38701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:47.817798 9448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:47.818022 9448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:57:47.822924 9448 server_base.cc:1034] running on GCE node
W20260926 06:57:47.822880 9456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:47.822861 9454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:47.822883 9453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:47.823338 9448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:47.823678 9448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:47.824831 9448 hybrid_clock.cc:648] HybridClock initialized: now 1790405867824804 us; error 38 us; skew 500 ppm
Sep 26 06:57:47 dist-test-slave-sfvt krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405867, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:47.827376 9448 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:47.827741 9448 webserver.cc:460] Webserver started at http://127.1.74.126:32973/ using document root <none> and password file <none>
I20260926 06:57:47.827929 9448 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:47.827970 9448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:47.828055 9448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:47.829028 9448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "e5c090d006634239866232924f462162"
format_stamp: "Formatted at 2026-09-26 06:57:47 on dist-test-slave-sfvt"
server_key: "6e834eaf3bf6f91c0e2c2b4d8e3b4801"
server_key_iv: "195b3698701d996386ce21eca18ee5a0"
server_key_version: "encryptionkey@0"
I20260926 06:57:47.829315 9448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "e5c090d006634239866232924f462162"
format_stamp: "Formatted at 2026-09-26 06:57:47 on dist-test-slave-sfvt"
server_key: "6e834eaf3bf6f91c0e2c2b4d8e3b4801"
server_key_iv: "195b3698701d996386ce21eca18ee5a0"
server_key_version: "encryptionkey@0"
I20260926 06:57:47.830552 9448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:47.831167 9463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:47.831349 9448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:47.831450 9448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "e5c090d006634239866232924f462162"
format_stamp: "Formatted at 2026-09-26 06:57:47 on dist-test-slave-sfvt"
server_key: "6e834eaf3bf6f91c0e2c2b4d8e3b4801"
server_key_iv: "195b3698701d996386ce21eca18ee5a0"
server_key_version: "encryptionkey@0"
I20260926 06:57:47.831528 9448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:47.850422 9448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:47.852836 9448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:47.853003 9448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:47.856760 9448 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:38701
I20260926 06:57:47.856804 9515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:38701 every 8 connection(s)
I20260926 06:57:47.857172 9448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:47.858049 9516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:57:47.860332 9516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: Bootstrap starting.
I20260926 06:57:47.860585 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 9448
I20260926 06:57:47.860702 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:57:47.860880 1321 external_mini_cluster.cc:1428] Setting key 44a9648511dcd33624060167a411622b
I20260926 06:57:47.861040 9516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: Neither blocks nor log segments found. Creating new log.
I20260926 06:57:47.861385 9516 log.cc:824] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:47.862114 9516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: No bootstrap required, opened a new log
I20260926 06:57:47.864424 9516 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } }
I20260926 06:57:47.864594 9516 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:57:47.864621 9516 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c090d006634239866232924f462162, State: Initialized, Role: FOLLOWER
I20260926 06:57:47.864763 9516 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } }
I20260926 06:57:47.864836 9516 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:47.864857 9516 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:47.864921 9516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:57:47.865515 9516 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } }
I20260926 06:57:47.865613 9516 leader_election.cc:302] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5c090d006634239866232924f462162; no voters:
I20260926 06:57:47.865762 9516 leader_election.cc:288] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:57:47.865829 9521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:57:47.865967 9521 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 1 LEADER]: Becoming Leader. State: Replica: e5c090d006634239866232924f462162, State: Running, Role: LEADER
I20260926 06:57:47.865988 9516 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:47.866060 9521 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } }
I20260926 06:57:47.866338 9523 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5c090d006634239866232924f462162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } } }
I20260926 06:57:47.866480 9523 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:47.866665 9521 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c090d006634239866232924f462162. Latest consensus state: current_term: 1 leader_uuid: "e5c090d006634239866232924f462162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } } }
I20260926 06:57:47.866730 9521 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: This master's current role is: LEADER
W20260926 06:57:47.867865 9536 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:57:47.868088 9536 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:57:47.868204 9529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:47.868362 9529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 06:57:47 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:47.869985 9529 catalog_manager.cc:1347] Generated new cluster ID: 79fee8c0d2b44c5a99bf282b67caea9e
I20260926 06:57:47.870035 9529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:47.873613 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:47.861378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.1.74.126:38701)
0926 06:57:47.861554 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:47.861563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:47.861597 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:57:47.862158 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:47.862167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:47.862231 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:47.862381 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:47.862955 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:47.864424 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:47.865338 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:47.865965 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:47.866416 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:47.869051 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:47.869055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:47.869055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:47.869066 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:47.871985 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:47.872351 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:47.872354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:47.872355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:47.872440 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:47.872643 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:47.872644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:47.872645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:47.872865 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:47.873026 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:47.873140 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 06:57:47.873335 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260926 06:57:47.883374 9529 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:57:47.883862 9529 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:47.890308 9529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: Generated new TSK 0
I20260926 06:57:47.890493 9529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:57:47.922173 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:38701
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:48.019253 9542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:48.019500 9542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:48.019523 9542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:48.020699 9542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:48.020737 9542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:48.020764 9542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:48.022181 9542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:38701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:48.022429 9542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:48.022647 9542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:48.027542 9548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:48.027559 9550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:48.027643 9542 server_base.cc:1034] running on GCE node
W20260926 06:57:48.027559 9547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:48.028028 9542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:48.028214 9542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:48.029353 9542 hybrid_clock.cc:648] HybridClock initialized: now 1790405868029329 us; error 39 us; skew 500 ppm
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:48.031737 9542 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:57:48.032163 9542 webserver.cc:460] Webserver started at http://127.1.74.65:33583/ using document root <none> and password file <none>
I20260926 06:57:48.032351 9542 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:48.032392 9542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:48.032477 9542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:48.033422 9542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "68a306510dcb4f75a91fca39530e07e6"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "328b8ec17019029a68eb79bde58d3459"
server_key_iv: "fc47764b4679388702dd98f5a9c8450a"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.033721 9542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "68a306510dcb4f75a91fca39530e07e6"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "328b8ec17019029a68eb79bde58d3459"
server_key_iv: "fc47764b4679388702dd98f5a9c8450a"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.035074 9542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:57:48.035924 9557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.036139 9542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:48.036228 9542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "68a306510dcb4f75a91fca39530e07e6"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "328b8ec17019029a68eb79bde58d3459"
server_key_iv: "fc47764b4679388702dd98f5a9c8450a"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.036284 9542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:48.056301 9542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:48.058683 9542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:48.058797 9542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:48.058983 9542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:48.059370 9542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:48.059407 9542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.059430 9542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:48.059473 9542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.065323 9542 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:45707
I20260926 06:57:48.065397 9670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:45707 every 8 connection(s)
I20260926 06:57:48.065730 9542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:57:48.067229 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 9542
I20260926 06:57:48.067390 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:57:48.067598 1321 external_mini_cluster.cc:1428] Setting key 18a1a4eb5a3328b042c15397cfa71e73
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
I20260926 06:57:48.077200 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.066612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:39843 (local address 127.1.74.126:38701)
0926 06:57:48.066695 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:48.066697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:48.066899 (+ 202us) server_negotiation.cc:407] Connection header received
0926 06:57:48.067740 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:48.067742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:48.067754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:48.067804 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:48.068917 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.069708 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.071373 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.071586 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.071645 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.074114 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:48.074115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:48.074115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:48.074124 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:48.075677 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:48.076126 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:48.076127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:48.076128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:48.076189 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:48.076466 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:48.076467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:48.076467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:48.076599 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:48.076694 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:48.077001 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 06:57:48.077101 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:57:48.077389 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.066602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:39843)
0926 06:57:48.066779 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:48.066814 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:48.067606 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:48.067893 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:48.067897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:48.068629 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:48.068820 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.068824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.069813 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.069814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.071291 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.071294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.071687 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.071688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.071851 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.072385 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:48.072397 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:48.074015 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:48.075776 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:48.075783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:48.075785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:48.076050 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:48.076257 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:48.076257 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:48.076258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:48.076369 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:48.076689 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:48.076696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:48.076864 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 06:57:48.076970 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 06:57:48.077142 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260926 06:57:48.077952 9671 heartbeater.cc:347] Connected to a master server at 127.1.74.126:38701
I20260926 06:57:48.078055 9671 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:48.078200 9671 heartbeater.cc:511] Master 127.1.74.126:38701 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
I20260926 06:57:48.078754 9480 ts_manager.cc:194] Registered new tserver with Master: 68a306510dcb4f75a91fca39530e07e6 (127.1.74.65:45707)
I20260926 06:57:48.079706 9480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:39843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:57:48.094663 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:38701
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:48.174155 9676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:48.174391 9676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:48.174430 9676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:48.175618 9676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:48.175683 9676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:48.175732 9676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:48.177156 9676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:38701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:48.177436 9676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:48.177661 9676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:48.182803 9682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:48.182781 9684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:48.182798 9676 server_base.cc:1034] running on GCE node
W20260926 06:57:48.182783 9681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:48.183235 9676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:48.183554 9676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:48.184700 9676 hybrid_clock.cc:648] HybridClock initialized: now 1790405868184684 us; error 27 us; skew 500 ppm
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:48.187186 9676 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:57:48.187613 9676 webserver.cc:460] Webserver started at http://127.1.74.66:37059/ using document root <none> and password file <none>
I20260926 06:57:48.187805 9676 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:48.187844 9676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:48.187928 9676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:48.188951 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "07c37c0bf2714ce4a43583b2c67b783b"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "257423c1090810d73bd9fb29c76592a1"
server_key_iv: "dd3e64504137fa79910ab5e0727852db"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.189265 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "07c37c0bf2714ce4a43583b2c67b783b"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "257423c1090810d73bd9fb29c76592a1"
server_key_iv: "dd3e64504137fa79910ab5e0727852db"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.190531 9676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:57:48.191202 9691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.191382 9676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:57:48.191484 9676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "07c37c0bf2714ce4a43583b2c67b783b"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "257423c1090810d73bd9fb29c76592a1"
server_key_iv: "dd3e64504137fa79910ab5e0727852db"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.191558 9676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:48.217537 9676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:48.220057 9676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:48.220227 9676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:48.220475 9676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:48.220801 9676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:48.220855 9676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.220896 9676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:48.220923 9676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.226608 9676 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:37359
I20260926 06:57:48.226707 9804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:37359 every 8 connection(s)
I20260926 06:57:48.227011 9676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:48.229250 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 9676
I20260926 06:57:48.229374 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:57:48.229547 1321 external_mini_cluster.cc:1428] Setting key 0f5e09eb23223afd11f3d103ed4fb88b
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:48.237239 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.227784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:49137 (local address 127.1.74.126:38701)
0926 06:57:48.227844 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:48.227846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:48.228038 (+ 192us) server_negotiation.cc:407] Connection header received
0926 06:57:48.228609 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:48.228610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:48.228622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:48.228680 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:48.229977 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.230480 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.231565 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.231772 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.231858 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.234305 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:48.234306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:48.234306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:48.234313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:48.235771 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:48.236297 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:48.236297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:48.236298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:48.236346 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:48.236623 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:48.236624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:48.236624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:48.236725 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:48.236773 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:48.237107 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 06:57:48.237168 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:57:48.237576 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.227769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:49137)
0926 06:57:48.227922 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:48.227957 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:48.228506 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:48.228760 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:48.228766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:48.229551 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:48.229878 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.229885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.230585 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.230586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.231469 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.231471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.231865 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.231866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.231989 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.232489 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:48.232497 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:48.234211 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:48.235860 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:48.235867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:48.235869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:48.236207 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:48.236413 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:48.236414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:48.236415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:48.236556 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:48.236793 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:48.236800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:48.236966 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 06:57:48.237094 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 06:57:48.237273 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
I20260926 06:57:48.238140 9805 heartbeater.cc:347] Connected to a master server at 127.1.74.126:38701
I20260926 06:57:48.238253 9805 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:48.238433 9805 heartbeater.cc:511] Master 127.1.74.126:38701 requested a full tablet report, sending...
I20260926 06:57:48.238898 9480 ts_manager.cc:194] Registered new tserver with Master: 07c37c0bf2714ce4a43583b2c67b783b (127.1.74.66:37359)
I20260926 06:57:48.239377 9480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:49137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:57:48.256464 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:38701
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:48.336047 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:48.336244 9810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:48.336266 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:48.337347 9810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:48.337383 9810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:48.337409 9810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:48.338718 9810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:38701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:48.338992 9810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:48.339190 9810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:48.344162 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:48.344192 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:48.344172 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:48.344344 9810 server_base.cc:1034] running on GCE node
I20260926 06:57:48.344540 9810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:48.344775 9810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:48.345907 9810 hybrid_clock.cc:648] HybridClock initialized: now 1790405868345899 us; error 26 us; skew 500 ppm
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:48.348258 9810 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:57:48.348616 9810 webserver.cc:460] Webserver started at http://127.1.74.67:40563/ using document root <none> and password file <none>
I20260926 06:57:48.348810 9810 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:48.348852 9810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:48.348935 9810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:57:48.349879 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "876ccbe205ff4f7f9de2b0efcefb4ef6"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "ee08acc9dad28689385a4f2321c6b4dd"
server_key_iv: "b24fbe67994661b5b21fab1f828e17c8"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.350157 9810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "876ccbe205ff4f7f9de2b0efcefb4ef6"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "ee08acc9dad28689385a4f2321c6b4dd"
server_key_iv: "b24fbe67994661b5b21fab1f828e17c8"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.351377 9810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:57:48.352039 9825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.352202 9810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.352267 9810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "876ccbe205ff4f7f9de2b0efcefb4ef6"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "ee08acc9dad28689385a4f2321c6b4dd"
server_key_iv: "b24fbe67994661b5b21fab1f828e17c8"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.352316 9810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:48.367352 9810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:48.369794 9810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:48.369939 9810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:48.370178 9810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:48.370491 9810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:48.370525 9810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.370548 9810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:48.370560 9810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.376410 9810 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:39667
I20260926 06:57:48.376487 9938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:39667 every 8 connection(s)
I20260926 06:57:48.376811 9810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:57:48.380851 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 9810
I20260926 06:57:48.380956 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:57:48.381143 1321 external_mini_cluster.cc:1428] Setting key c42286e3f0f8aca3127065090bec9ef7
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:57:48.385908 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.377571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:37423 (local address 127.1.74.126:38701)
0926 06:57:48.377627 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:48.377628 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:48.377830 (+ 202us) server_negotiation.cc:407] Connection header received
0926 06:57:48.378355 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:48.378357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:48.378369 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:48.378435 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:48.379407 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.379904 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.381115 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.381447 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.381516 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.383672 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:48.383673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:48.383674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:48.383680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:48.384824 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:48.385155 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:48.385155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:48.385156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:48.385201 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:48.385432 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:48.385433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:48.385433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:48.385526 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:48.385574 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:48.385790 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 06:57:48.385849 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:57:48.386076 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.377551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:37423)
0926 06:57:48.377715 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:48.377749 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:48.378255 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:48.378502 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:48.378506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:48.379067 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:48.379304 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.379310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.380000 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.380001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.381017 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.381020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.381553 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.381554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.381717 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.382267 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:48.382275 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:48.383593 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:57:48.384887 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:48.384893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:48.384894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:48.385092 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:48.385275 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:57:48.385276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:57:48.385276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:57:48.385372 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:57:48.385577 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:57:48.385581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:57:48.385687 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 06:57:48.385766 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:57:48.385878 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260926 06:57:48.386591 9939 heartbeater.cc:347] Connected to a master server at 127.1.74.126:38701
I20260926 06:57:48.386668 9939 heartbeater.cc:464] Registering TS with master...
I20260926 06:57:48.386852 9939 heartbeater.cc:511] Master 127.1.74.126:38701 requested a full tablet report, sending...
I20260926 06:57:48.387212 9480 ts_manager.cc:194] Registered new tserver with Master: 876ccbe205ff4f7f9de2b0efcefb4ef6 (127.1.74.67:39667)
I20260926 06:57:48.387588 9480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:37423
I20260926 06:57:48.392710 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab.
Entry for principal oryx/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab.
WARNING: no policy specified for oryx/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab.
Entry for principal oryx/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab.
WARNING: no policy specified for oryx/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab.
Entry for principal oryx/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab.
I20260926 06:57:48.462565 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 9542
I20260926 06:57:48.467113 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 9676
I20260926 06:57:48.471454 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 9810
I20260926 06:57:48.475416 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 9448
I20260926 06:57:48.480254 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:38701
--webserver_interface=127.1.74.126
--webserver_port=32973
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:38701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:57:48.557821 9948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:48.558059 9948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:48.558097 9948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:48.559239 9948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:57:48.559346 9948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:48.559369 9948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:48.559403 9948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:57:48.559432 9948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:57:48.560920 9948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:38701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:38701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=32973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:48.561208 9948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:48.561443 9948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:48.566192 9954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:48.566270 9956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:48.566205 9953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:48.566376 9948 server_base.cc:1034] running on GCE node
I20260926 06:57:48.566659 9948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:48.566954 9948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:48.568135 9948 hybrid_clock.cc:648] HybridClock initialized: now 1790405868568111 us; error 33 us; skew 500 ppm
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405868, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:48.570595 9948 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:57:48.571012 9948 webserver.cc:460] Webserver started at http://127.1.74.126:32973/ using document root <none> and password file <none>
I20260926 06:57:48.571195 9948 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:48.571244 9948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:48.572551 9948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:57:48.573158 9963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.573340 9948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:57:48.573426 9948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "e5c090d006634239866232924f462162"
format_stamp: "Formatted at 2026-09-26 06:57:47 on dist-test-slave-sfvt"
server_key: "6e834eaf3bf6f91c0e2c2b4d8e3b4801"
server_key_iv: "195b3698701d996386ce21eca18ee5a0"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.573727 9948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:48.590327 9948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:48.592514 9948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:48.592619 9948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:48.596312 9948 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:38701
I20260926 06:57:48.596360 10015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:38701 every 8 connection(s)
I20260926 06:57:48.596710 9948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:57:48.598084 10016 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:57:48.598738 10016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: Bootstrap starting.
I20260926 06:57:48.604923 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 9948
I20260926 06:57:48.605194 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:45707
--local_ip_for_outbound_sockets=127.1.74.65
--tserver_master_addrs=127.1.74.126:38701
--webserver_port=33583
--webserver_interface=127.1.74.65
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.74.65
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:57:48.606947 10016 log.cc:824] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: Log is configured to *not* fsync() on all Append() calls
I20260926 06:57:48.608553 10016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:57:48.608836 10016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: Bootstrap complete.
I20260926 06:57:48.610956 10016 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } }
I20260926 06:57:48.611191 10016 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c090d006634239866232924f462162, State: Initialized, Role: FOLLOWER
I20260926 06:57:48.611400 10016 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } }
I20260926 06:57:48.611480 10016 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:57:48.611532 10016 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:57:48.611578 10016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:57:48.612510 10016 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } }
I20260926 06:57:48.612624 10016 leader_election.cc:302] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5c090d006634239866232924f462162; no voters:
I20260926 06:57:48.612831 10016 leader_election.cc:288] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:57:48.612860 10020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:57:48.612962 10020 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [term 2 LEADER]: Becoming Leader. State: Replica: e5c090d006634239866232924f462162, State: Running, Role: LEADER
I20260926 06:57:48.613068 10020 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } }
I20260926 06:57:48.613072 10016 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:57:48.613394 10022 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e5c090d006634239866232924f462162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } } }
I20260926 06:57:48.613504 10022 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:48.613670 10021 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c090d006634239866232924f462162. Latest consensus state: current_term: 2 leader_uuid: "e5c090d006634239866232924f462162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c090d006634239866232924f462162" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 38701 } } }
I20260926 06:57:48.613741 10021 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162 [sys.catalog]: This master's current role is: LEADER
I20260926 06:57:48.615010 10035 catalog_manager.cc:1259] Loaded cluster ID: 79fee8c0d2b44c5a99bf282b67caea9e
I20260926 06:57:48.615057 10035 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: loading cluster ID for follower catalog manager: success
I20260926 06:57:48.615908 10035 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: acquiring CA information for follower catalog manager: success
I20260926 06:57:48.616201 10035 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:57:48.616274 10024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:57:48.616417 10024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:57:48.616508 10024 catalog_manager.cc:1259] Loaded cluster ID: 79fee8c0d2b44c5a99bf282b67caea9e
I20260926 06:57:48.616545 10024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:57:48.616704 10024 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:57:48.616814 10024 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5c090d006634239866232924f462162: Loaded TSK: 0
I20260926 06:57:48.616946 10024 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:57:48.702754 10018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:48.702980 10018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:48.703020 10018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:48.704308 10018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:48.704371 10018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:48.704416 10018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:57:48.705860 10018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.65.keytab
--principal=oryx/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:45707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=33583
--tserver_master_addrs=127.1.74.126:38701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:48.706111 10018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:48.706337 10018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:48.711179 10042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:48.711182 10045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:48.711500 10018 server_base.cc:1034] running on GCE node
W20260926 06:57:48.711215 10043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:48.711790 10018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:48.712059 10018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:48.713209 10018 hybrid_clock.cc:648] HybridClock initialized: now 1790405868713197 us; error 26 us; skew 500 ppm
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:48.715862 10018 init.cc:377] Logged in from keytab as oryx/127.1.74.65@KRBTEST.COM (short username oryx)
I20260926 06:57:48.716341 10018 webserver.cc:460] Webserver started at http://127.1.74.65:33583/ using document root <none> and password file <none>
I20260926 06:57:48.716583 10018 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:48.716638 10018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:48.717989 10018 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:48.718627 10052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.718775 10018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:48.718869 10018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "68a306510dcb4f75a91fca39530e07e6"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "328b8ec17019029a68eb79bde58d3459"
server_key_iv: "fc47764b4679388702dd98f5a9c8450a"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.719182 10018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:48.740144 10018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:48.742286 10018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:48.742431 10018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:48.742692 10018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:48.743050 10018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:48.743095 10018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.743117 10018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:48.743162 10018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.750094 10018 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:45707
I20260926 06:57:48.750306 10165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:45707 every 8 connection(s)
I20260926 06:57:48.750530 10018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:48.758482 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.751294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:41593)
0926 06:57:48.751453 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:48.751494 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:48.751993 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:48.752409 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:48.752413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:48.752974 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:48.753151 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.753155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.754247 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.754248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.755029 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.755031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.755742 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.755743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.755872 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.756345 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:48.756353 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:48.758348 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
W20260926 06:57:48.758641 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:48.751331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:41593 (local address 127.1.74.126:38701)
0926 06:57:48.751566 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:48.751574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:48.751616 (+ 42us) server_negotiation.cc:407] Connection header received
0926 06:57:48.752186 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:48.752192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:48.752242 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:48.752376 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:48.753270 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.754143 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.755129 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.755640 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.755766 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.758461 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:41593: BlockingRecv error: recv got EOF from 127.1.74.65:41593 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
W20260926 06:57:48.758751 10166 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:38701 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.74.126:38701: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
I20260926 06:57:48.760313 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 10018
I20260926 06:57:48.760597 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:37359
--local_ip_for_outbound_sockets=127.1.74.66
--tserver_master_addrs=127.1.74.126:38701
--webserver_port=37059
--webserver_interface=127.1.74.66
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.74.66
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:48.763105 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.758990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:36833)
0926 06:57:48.759042 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:48.759054 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:48.759100 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:48.759197 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:48.759197 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:48.759370 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:48.759410 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.759411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.759973 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.759974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.760532 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.760534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.760972 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.760973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.761024 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.761428 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:48.761433 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:48.763063 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 06:57:48.763247 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:48.759059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:36833 (local address 127.1.74.126:38701)
0926 06:57:48.759114 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:48.759115 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:48.759122 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:48.759141 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:48.759142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:48.759150 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:48.759194 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:48.759473 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.759911 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.760615 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.760903 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.760974 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.763167 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:36833: BlockingRecv error: recv got EOF from 127.1.74.65:36833 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:48.767659 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.763424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:54715)
0926 06:57:48.763505 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:48.763514 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:48.763546 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:48.763657 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:48.763657 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:48.763765 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:48.763790 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.763791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.764697 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.764697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.765164 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.765166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.765648 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.765649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.765684 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.766090 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:48.766094 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:48.767616 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 06:57:48.767787 10166 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:57:48.767799 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:48.763522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:54715 (local address 127.1.74.126:38701)
0926 06:57:48.763557 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:48.763559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:48.763566 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:48.763588 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:48.763589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:48.763598 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:48.763654 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:48.763858 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.764621 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.765241 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.765573 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.765648 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.767726 (+ 2078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:54715: BlockingRecv error: recv got EOF from 127.1.74.65:54715 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 06:57:48.841476 10170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:48.841665 10170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:48.841688 10170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:48.842775 10170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:48.842816 10170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:48.842841 10170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:57:48.844327 10170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.66.keytab
--principal=oryx/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:37359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=37059
--tserver_master_addrs=127.1.74.126:38701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:48.844506 10170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:48.844671 10170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:57:48.849584 10170 server_base.cc:1034] running on GCE node
W20260926 06:57:48.849573 10178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:48.849545 10175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:48.849552 10176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:48.849985 10170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:48.850217 10170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:48.851311 10170 hybrid_clock.cc:648] HybridClock initialized: now 1790405868851301 us; error 27 us; skew 500 ppm
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:48.853626 10170 init.cc:377] Logged in from keytab as oryx/127.1.74.66@KRBTEST.COM (short username oryx)
I20260926 06:57:48.853999 10170 webserver.cc:460] Webserver started at http://127.1.74.66:37059/ using document root <none> and password file <none>
I20260926 06:57:48.854185 10170 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:48.854233 10170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:48.855535 10170 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:57:48.856050 10185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.856222 10170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.856307 10170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "07c37c0bf2714ce4a43583b2c67b783b"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "257423c1090810d73bd9fb29c76592a1"
server_key_iv: "dd3e64504137fa79910ab5e0727852db"
server_key_version: "encryptionkey@0"
I20260926 06:57:48.856705 10170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:48.900511 10170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:48.903242 10170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:48.903412 10170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:48.903667 10170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:48.904014 10170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:48.904070 10170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.904101 10170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:48.904136 10170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:48.910593 10170 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:37359
I20260926 06:57:48.910660 10298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:37359 every 8 connection(s)
I20260926 06:57:48.911007 10170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:57:48.913549 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 10170
I20260926 06:57:48.913847 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:39667
--local_ip_for_outbound_sockets=127.1.74.67
--tserver_master_addrs=127.1.74.126:38701
--webserver_port=40563
--webserver_interface=127.1.74.67
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.74.67
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:57:48.920396 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:48.911762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:59625 (local address 127.1.74.126:38701)
0926 06:57:48.911811 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:48.911813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:48.912023 (+ 210us) server_negotiation.cc:407] Connection header received
0926 06:57:48.912503 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:48.912504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:48.912517 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:48.912622 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:48.913740 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.914498 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.916399 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.916599 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.916657 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.920345 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:59625: BlockingRecv error: recv got EOF from 127.1.74.66:59625 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:57:48.920470 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.911745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:59625)
0926 06:57:48.911898 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:48.911932 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:48.912413 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:48.912676 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:48.912679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:48.913305 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:48.913570 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.913575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.914576 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.914577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.916128 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.916131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.916687 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.916688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.916811 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.917403 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:48.917412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:48.920214 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
W20260926 06:57:48.920722 10299 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:38701 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.74.126:38701: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:48.926561 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.921010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:35621)
0926 06:57:48.921149 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:48.921162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:48.921221 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:48.921360 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:48.921361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:48.921522 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:48.921582 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.921583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.922375 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.922376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.923390 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.923393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.923755 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.923756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.923830 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.924254 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:48.924260 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:48.926485 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119}
W20260926 06:57:48.926684 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:48.921094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:35621 (local address 127.1.74.126:38701)
0926 06:57:48.921161 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:48.921163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:48.921203 (+ 40us) server_negotiation.cc:407] Connection header received
0926 06:57:48.921294 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:48.921295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:48.921303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:48.921354 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:48.921673 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.922247 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.923471 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.923677 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.923741 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.926589 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:35621: BlockingRecv error: recv got EOF from 127.1.74.66:35621 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:48 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:48.931556 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:48.927010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:36641)
0926 06:57:48.927050 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:48.927059 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:48.927091 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:48.927286 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:48.927287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:48.927422 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:48.927454 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.927455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.928075 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.928076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.928887 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:48.928889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.929307 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.929308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:48.929361 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.929733 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:48.929738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:48.931488 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 06:57:48.931607 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:48.927085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:36641 (local address 127.1.74.126:38701)
0926 06:57:48.927143 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:48.927148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:48.927156 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:48.927173 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:48.927173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:48.927181 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:48.927229 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:48.927541 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.928006 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.929029 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:48.929232 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:48.929295 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:48.931563 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:36641: BlockingRecv error: recv got EOF from 127.1.74.66:36641 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 06:57:48.931784 10299 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:57:48.999691 10302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:57:48.999933 10302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:57:48.999969 10302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:57:49.001134 10302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:57:49.001205 10302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:57:49.001246 10302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:57:49.002686 10302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/krb5kdc/oryx_127.1.74.67.keytab
--principal=oryx/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:39667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=40563
--tserver_master_addrs=127.1.74.126:38701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:57:49.002948 10302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:57:49.003178 10302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:57:49.008284 10307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:49.008320 10308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:57:49.008414 10310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:57:49.008415 10302 server_base.cc:1034] running on GCE node
I20260926 06:57:49.008728 10302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:57:49.008921 10302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:57:49.010085 10302 hybrid_clock.cc:648] HybridClock initialized: now 1790405869010062 us; error 37 us; skew 500 ppm
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405869, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:57:49.012521 10302 init.cc:377] Logged in from keytab as oryx/127.1.74.67@KRBTEST.COM (short username oryx)
I20260926 06:57:49.013015 10302 webserver.cc:460] Webserver started at http://127.1.74.67:40563/ using document root <none> and password file <none>
I20260926 06:57:49.013247 10302 fs_manager.cc:359] Metadata directory not provided
I20260926 06:57:49.013332 10302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:57:49.014757 10302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:57:49.015379 10317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:49.015547 10302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:49.015641 10302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "876ccbe205ff4f7f9de2b0efcefb4ef6"
format_stamp: "Formatted at 2026-09-26 06:57:48 on dist-test-slave-sfvt"
server_key: "ee08acc9dad28689385a4f2321c6b4dd"
server_key_iv: "b24fbe67994661b5b21fab1f828e17c8"
server_key_version: "encryptionkey@0"
I20260926 06:57:49.016000 10302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:57:49.031701 10302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:57:49.033519 10302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:57:49.033682 10302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:57:49.033912 10302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:57:49.034215 10302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:57:49.034269 10302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:49.034309 10302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:57:49.034337 10302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:57:49.039995 10302 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:39667
I20260926 06:57:49.040042 10430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:39667 every 8 connection(s)
I20260926 06:57:49.040390 10302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:57:49.047401 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:49.041068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:48943 (local address 127.1.74.126:38701)
0926 06:57:49.041113 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:49.041115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:49.041328 (+ 213us) server_negotiation.cc:407] Connection header received
0926 06:57:49.041791 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:49.041792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:49.041802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:49.041871 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:49.042785 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.043437 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.044363 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.044541 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.044591 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.047347 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:48943: BlockingRecv error: recv got EOF from 127.1.74.67:48943 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:57:49.047428 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:49.041039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:48943)
0926 06:57:49.041186 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:49.041226 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:49.041687 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:49.041926 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:49.041930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:49.042506 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:49.042691 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.042695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.043512 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.043512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.044283 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.044284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.044609 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.044610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.044704 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.045103 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:49.045111 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:49.047226 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
W20260926 06:57:49.047657 10431 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:38701 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.74.126:38701: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
I20260926 06:57:49.049873 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 10302
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:49.052340 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:49.047860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:45477)
0926 06:57:49.047912 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:49.047924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:49.047967 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:49.048209 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:49.048210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:49.048351 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:49.048390 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.048391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.049008 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.049009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.049497 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.049498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.049973 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.049975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.050044 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.050454 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:49.050459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:49.052274 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:57:49.052419 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:49.048053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:45477 (local address 127.1.74.126:38701)
0926 06:57:49.048105 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:49.048106 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:49.048113 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:49.048128 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:49.048129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:49.048136 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:49.048193 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:49.048465 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.048916 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.049585 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.049878 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.049953 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.052349 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:45477: BlockingRecv error: recv got EOF from 127.1.74.67:45477 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:49.057631 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:49.052557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:58979)
0926 06:57:49.052616 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:49.052625 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:49.052656 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:49.052794 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:49.052795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:49.052954 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:49.052997 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.052998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.053895 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.053896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.054720 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.054723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.055164 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.055165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.055218 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.055617 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:49.055621 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:49.057573 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 06:57:49.057828 10169 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:49.052610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:58979 (local address 127.1.74.126:38701)
0926 06:57:49.052640 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:49.052641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:49.052649 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:49.052710 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:49.052711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:49.052718 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:49.052771 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:49.053062 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.053806 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.054815 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.055078 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.055149 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.057651 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:58979: BlockingRecv error: recv got EOF from 127.1.74.67:58979 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 06:57:49.057868 10431 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:57:49.058362 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:49.050256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.1.74.126:38701)
0926 06:57:49.050382 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:49.050383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:49.050575 (+ 192us) server_negotiation.cc:407] Connection header received
0926 06:57:49.050633 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:49.050633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:49.050643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:49.050693 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:49.051379 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.052039 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.053151 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.053436 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.053503 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.054839 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:57:49.054844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:57:49.054845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:57:49.054856 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:57:49.056786 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:49.057221 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:49.057224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:49.057225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:49.057359 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:57:49.057549 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:57:49.057550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:57:49.057550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:57:49.057847 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:57:49.058003 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 06:57:49.058208 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 06:57:49.058291 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":84,"threads_started":1}
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:49.777055 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:49.768161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:42951)
0926 06:57:49.768294 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:49.768304 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:49.768356 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:49.768475 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:49.768476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:49.768633 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:49.768681 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.768682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.769327 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.769328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.769927 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.769929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.770260 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.770261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.770305 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.775085 (+ 4780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:49.775093 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:49.776985 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260926 06:57:49.777176 10437 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:49.768214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:42951 (local address 127.1.74.126:38701)
0926 06:57:49.768348 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:49.768349 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:49.768358 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:49.768402 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:49.768402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:49.768411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:49.768465 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:49.768754 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.769254 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.770000 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.770186 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.770236 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.777090 (+ 6854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:42951: BlockingRecv error: recv got EOF from 127.1.74.65:42951 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:49 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:49.936964 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:49.932189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:53255)
0926 06:57:49.932437 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:49.932448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:49.932533 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:49.932697 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:49.932697 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:49.932886 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:49.932933 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.932935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.933606 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.933607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.934157 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:49.934158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.934540 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.934541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:49.934596 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.935111 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:49.935116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:49.936879 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":94,"threads_started":1}
W20260926 06:57:49.937018 10437 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:49.932300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:53255 (local address 127.1.74.126:38701)
0926 06:57:49.932388 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:49.932390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:49.932516 (+ 126us) server_negotiation.cc:407] Connection header received
0926 06:57:49.932611 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:49.932612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:49.932625 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:49.932684 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:49.933017 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.933519 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.934246 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:49.934454 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:49.934539 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:49.936972 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:53255: BlockingRecv error: recv got EOF from 127.1.74.66:53255 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 06:57:50 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:50 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:50.063040 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:50.058237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:43327)
0926 06:57:50.058420 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:50.058433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:50.058545 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:50.058707 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:50.058708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:50.058900 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:50.058947 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:50.058949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.059675 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.059676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:50.060279 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:50.060281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.060640 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.060641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:50.060709 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:50.061262 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:50.061267 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:50.062967 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
W20260926 06:57:50.063167 10437 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:50.058342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:43327 (local address 127.1.74.126:38701)
0926 06:57:50.058438 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:50.058440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:50.058526 (+ 86us) server_negotiation.cc:407] Connection header received
0926 06:57:50.058640 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:50.058641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:50.058652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:50.058703 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:50.059029 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.059595 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.060373 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.060563 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.060643 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:50.063067 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:43327: BlockingRecv error: recv got EOF from 127.1.74.67:43327 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 06:57:50 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:50 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:50.782326 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:50.777596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:46103)
0926 06:57:50.777754 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:50.777766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:50.777874 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:50.778095 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:50.778096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:50.778356 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:50.778416 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:50.778418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.779162 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.779163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:50.779736 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:50.779738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.780053 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.780054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:50.780095 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:50.780592 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:50.780597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:50.782256 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
W20260926 06:57:50.782398 10441 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:50.777670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:46103 (local address 127.1.74.126:38701)
0926 06:57:50.777780 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:50.777782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:50.777876 (+ 94us) server_negotiation.cc:407] Connection header received
0926 06:57:50.777952 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:50.777954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:50.777970 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:50.778056 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:50.778520 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.779086 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.779807 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.779989 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.780041 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:50.782345 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:46103: BlockingRecv error: recv got EOF from 127.1.74.65:46103 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
Sep 26 06:57:50 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:50 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:50.942353 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:50.937476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:48115)
0926 06:57:50.937660 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:50.937672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:50.937759 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:50.937912 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:50.937913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:50.938117 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:50.938175 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:50.938178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.939070 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.939071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:50.939709 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:50.939712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.940069 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.940070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:50.940118 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:50.940550 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:50.940554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:50.942293 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
W20260926 06:57:50.942510 10441 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:50.937585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:48115 (local address 127.1.74.126:38701)
0926 06:57:50.937655 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:50.937658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:50.937739 (+ 81us) server_negotiation.cc:407] Connection header received
0926 06:57:50.937826 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:50.937827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:50.937844 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:50.937908 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:50.938306 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.938962 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.939788 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:50.939968 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:50.940021 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:50.942424 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:48115: BlockingRecv error: recv got EOF from 127.1.74.66:48115 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 06:57:51 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:51 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:51.067904 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:51.063579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:52377)
0926 06:57:51.063713 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:51.063723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:51.063778 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:51.063905 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:51.063906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:51.064069 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:51.064115 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:51.064117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.064724 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.064725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:51.065293 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:51.065294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.065596 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.065596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:51.065641 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:51.066072 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:51.066077 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:51.067811 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
W20260926 06:57:51.067945 10441 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:51.063646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:52377 (local address 127.1.74.126:38701)
0926 06:57:51.063719 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:51.063722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:51.063780 (+ 58us) server_negotiation.cc:407] Connection header received
0926 06:57:51.063822 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:51.063822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:51.063835 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:51.063887 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:51.064176 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.064654 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.065366 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.065536 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.065584 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:51.067860 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:52377: BlockingRecv error: recv got EOF from 127.1.74.67:52377 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 06:57:51 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:51 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:51.791913 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:51.782813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:59847)
0926 06:57:51.782935 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:51.782948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:51.783022 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:51.783164 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:51.783165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:51.783392 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:51.783461 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:51.783463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.784133 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.784134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:51.784662 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:51.784664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.784985 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.784985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:51.785023 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:51.789934 (+ 4911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:51.789939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:51.791802 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":52,"threads_started":1}
W20260926 06:57:51.791955 10445 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:51.782886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:59847 (local address 127.1.74.126:38701)
0926 06:57:51.783043 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:51.783045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:51.783053 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:51.783076 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:51.783076 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:51.783085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:51.783143 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:51.783546 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.784051 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.784733 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.784924 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.784973 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:51.791883 (+ 6910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:59847: BlockingRecv error: recv got EOF from 127.1.74.65:59847 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
Sep 26 06:57:51 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:51 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:51.947366 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:51.942873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:39187)
0926 06:57:51.943026 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:51.943037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:51.943100 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:51.943230 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:51.943230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:51.943403 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:51.943462 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:51.943464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.944135 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.944136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:51.944688 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:51.944691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.945021 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.945021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:51.945058 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:51.945567 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:51.945572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:51.947248 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
W20260926 06:57:51.947424 10445 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:51.942945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:39187 (local address 127.1.74.126:38701)
0926 06:57:51.943011 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:51.943013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:51.943102 (+ 89us) server_negotiation.cc:407] Connection header received
0926 06:57:51.943142 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:51.943143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:51.943155 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:51.943207 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:51.943535 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.944029 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.944770 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:51.944959 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:51.945010 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:51.947379 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:39187: BlockingRecv error: recv got EOF from 127.1.74.66:39187 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 06:57:52 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:52 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:52.072961 10447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:52.068382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:33457)
0926 06:57:52.068542 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:52.068555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:52.068620 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:52.068756 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:52.068757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:52.068949 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:52.068995 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:52.068997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.069670 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.069672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:52.070226 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:52.070228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.070589 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.070590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:52.070645 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:52.071065 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:52.071069 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:52.072853 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
W20260926 06:57:52.072997 10445 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:52.068463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:33457 (local address 127.1.74.126:38701)
0926 06:57:52.068546 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:52.068548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:52.068600 (+ 52us) server_negotiation.cc:407] Connection header received
0926 06:57:52.068680 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:52.068681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:52.068695 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:52.068741 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:52.069075 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.069581 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.070321 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.070515 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.070592 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:52.072933 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:33457: BlockingRecv error: recv got EOF from 127.1.74.67:33457 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 06:57:52 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:52 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:52.796999 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:52.792409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:50609)
0926 06:57:52.792552 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:52.792563 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:52.792609 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:52.792830 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:52.792832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:52.792989 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:52.793039 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:52.793040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.793696 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.793697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:52.794248 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:52.794251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.794572 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.794573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:52.794611 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:52.795135 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:52.795140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:52.796906 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
W20260926 06:57:52.797043 10449 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:52.792497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:50609 (local address 127.1.74.126:38701)
0926 06:57:52.792620 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:52.792622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:52.792629 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:57:52.792651 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:52.792651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:52.792661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:52.792722 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:52.793128 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.793621 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.794326 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.794508 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.794558 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:52.796975 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:50609: BlockingRecv error: recv got EOF from 127.1.74.65:50609 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
Sep 26 06:57:52 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:52 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:52.952807 10450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:52.947893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:42069)
0926 06:57:52.948090 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:52.948104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:52.948197 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:52.948326 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:52.948327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:52.948510 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:52.948555 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:52.948556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.949255 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.949256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:52.949793 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:52.949795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.950193 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.950194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:52.950257 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:52.950720 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:52.950725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:52.952713 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":115,"threads_started":1}
W20260926 06:57:52.952850 10449 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:52.948026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:42069 (local address 127.1.74.126:38701)
0926 06:57:52.948094 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:52.948097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:52.948179 (+ 82us) server_negotiation.cc:407] Connection header received
0926 06:57:52.948255 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:52.948256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:52.948269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:52.948322 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:52.948638 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.949166 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.949887 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:52.950097 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:52.950193 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:52.952787 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:42069: BlockingRecv error: recv got EOF from 127.1.74.66:42069 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 06:57:53 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:53 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:53.078261 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:53.073470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:53549)
0926 06:57:53.073644 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:53.073655 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:53.073725 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:53.073857 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:53.073857 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:53.074013 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:53.074058 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:53.074059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.074694 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.074696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:53.075245 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:53.075247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.075610 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.075611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:53.075657 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:53.076300 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:53.076306 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:53.078199 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
W20260926 06:57:53.078393 10449 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:53.073572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:53549 (local address 127.1.74.126:38701)
0926 06:57:53.073617 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:53.073619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:53.073744 (+ 125us) server_negotiation.cc:407] Connection header received
0926 06:57:53.073781 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:53.073781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:53.073793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:53.073832 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:53.074124 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.074600 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.075356 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.075536 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.075588 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:53.078290 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:53549: BlockingRecv error: recv got EOF from 127.1.74.67:53549 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 06:57:53 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:53 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:53.802412 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:53.797590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:52291)
0926 06:57:53.797716 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:53.797729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:53.797787 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:53.797945 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:53.797945 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:53.798100 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:53.798148 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:53.798150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.798788 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.798789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:53.799524 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:53.799527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.799916 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.799918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:53.799959 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:53.800397 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:53.800402 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:53.802350 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260926 06:57:53.802609 10453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:53.797666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:52291 (local address 127.1.74.126:38701)
0926 06:57:53.797799 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:53.797801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:53.797810 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:53.797835 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:53.797835 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:53.797845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:53.797914 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:53.798213 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.798714 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.799626 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.799846 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.799901 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:53.802488 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:52291: BlockingRecv error: recv got EOF from 127.1.74.65:52291 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
Sep 26 06:57:53 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:53 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:53.957895 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:53.953289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:50071)
0926 06:57:53.953421 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:53.953434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:53.953494 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:53.953659 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:53.953660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:53.953835 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:53.953880 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:53.953882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.954559 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.954560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:53.955095 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:53.955097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.955550 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.955551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:53.955613 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:53.956097 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:53.956102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:53.957837 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":54,"threads_started":1}
W20260926 06:57:53.957973 10453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:53.953442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:50071 (local address 127.1.74.126:38701)
0926 06:57:53.953507 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:53.953509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:53.953540 (+ 31us) server_negotiation.cc:407] Connection header received
0926 06:57:53.953567 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:53.953568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:53.953579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:53.953652 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:53.953970 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.954469 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.955189 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:53.955459 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:53.955550 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:53.957925 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:50071: BlockingRecv error: recv got EOF from 127.1.74.66:50071 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 06:57:54 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:54 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:54.083469 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:54.078778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:52863)
0926 06:57:54.078919 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:54.078930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:54.078989 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:54.079122 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:54.079123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:54.079304 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:54.079351 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:54.079353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.079979 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.079979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:54.080491 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:54.080492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.080805 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.080806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:54.080845 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:54.081357 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:54.081361 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:54.083410 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
W20260926 06:57:54.083613 10453 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:54.078836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:52863 (local address 127.1.74.126:38701)
0926 06:57:54.078903 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:54.078905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:54.079023 (+ 118us) server_negotiation.cc:407] Connection header received
0926 06:57:54.079048 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:54.079049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:54.079061 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:54.079102 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:54.079416 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.079907 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.080569 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.080742 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.080793 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:54.083498 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:52863: BlockingRecv error: recv got EOF from 127.1.74.67:52863 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 06:57:54 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:54 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:54.808130 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:54.803012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:42785)
0926 06:57:54.803141 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:54.803155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:54.803228 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:54.803551 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:54.803553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:54.803783 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:54.803836 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:54.803838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.804508 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.804509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:54.805030 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:54.805032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.805398 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.805399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:54.805453 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:54.806015 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:54.806020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:54.808033 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
W20260926 06:57:54.808178 10457 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:54.803102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:42785 (local address 127.1.74.126:38701)
0926 06:57:54.803242 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:54.803244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:54.803269 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:57:54.803362 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:54.803363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:54.803373 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:54.803433 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:54.803929 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.804431 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.805112 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.805300 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.805356 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:54.808109 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:42785: BlockingRecv error: recv got EOF from 127.1.74.65:42785 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
Sep 26 06:57:54 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:54 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:54.963361 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:54.958451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:55931)
0926 06:57:54.958598 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:54.958612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:54.958677 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:54.958843 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:54.958844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:54.959052 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:54.959123 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:54.959125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.959841 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.959842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:54.960522 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:54.960524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.960896 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.960896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:54.960934 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:54.961483 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:54.961488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:54.963303 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
W20260926 06:57:54.963469 10457 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:54.958500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:55931 (local address 127.1.74.126:38701)
0926 06:57:54.958600 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:54.958602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:54.958677 (+ 75us) server_negotiation.cc:407] Connection header received
0926 06:57:54.958733 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:54.958734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:54.958745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:54.958832 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:54.959185 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.959740 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.960595 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:54.960829 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:54.960883 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:54.963396 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:55931: BlockingRecv error: recv got EOF from 127.1.74.66:55931 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 06:57:55 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:55 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:55.088230 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:55.083883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:60467)
0926 06:57:55.083991 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:55.084002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:55.084065 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:55.084169 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:55.084170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:55.084322 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:55.084367 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:55.084369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.084974 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.084975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:55.085475 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:55.085476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.085795 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.085795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:55.085833 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:55.086314 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:55.086318 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:55.088173 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
W20260926 06:57:55.088369 10457 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:55.083957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:60467 (local address 127.1.74.126:38701)
0926 06:57:55.084001 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:55.084003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:55.084053 (+ 50us) server_negotiation.cc:407] Connection header received
0926 06:57:55.084108 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:55.084108 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:55.084120 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:55.084164 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:55.084428 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.084911 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.085555 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.085731 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.085783 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:55.088272 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:60467: BlockingRecv error: recv got EOF from 127.1.74.67:60467 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 06:57:55 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:55 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:55.813246 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:55.808629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:43617)
0926 06:57:55.808820 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:55.808830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:55.808884 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:55.809023 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:55.809023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:55.809193 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:55.809239 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:55.809241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.809871 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.809872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:55.810397 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:55.810398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.810736 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.810737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:55.810783 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:55.811218 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:55.811223 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:55.813175 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":100,"threads_started":1}
W20260926 06:57:55.813447 10461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:55.808765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:43617 (local address 127.1.74.126:38701)
0926 06:57:55.808903 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:55.808905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:55.808913 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:57:55.808935 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:55.808935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:55.808945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:55.809002 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:55.809312 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.809797 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.810473 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.810656 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.810706 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:55.813322 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:43617: BlockingRecv error: recv got EOF from 127.1.74.65:43617 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
Sep 26 06:57:55 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:55 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:55.968317 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:55.963896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:52691)
0926 06:57:55.964020 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:55.964039 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:55.964098 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:55.964215 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:55.964216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:55.964368 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:55.964414 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:55.964416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.965057 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.965059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:55.965575 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:55.965577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.965930 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.965930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:55.965970 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:55.966422 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:55.966426 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:55.968223 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
W20260926 06:57:55.968361 10461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:55.963961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:52691 (local address 127.1.74.126:38701)
0926 06:57:55.964036 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:55.964040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:55.964100 (+ 60us) server_negotiation.cc:407] Connection header received
0926 06:57:55.964146 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:55.964147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:55.964157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:55.964203 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:55.964480 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.964968 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.965663 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:55.965865 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:55.965915 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:55.968299 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:52691: BlockingRecv error: recv got EOF from 127.1.74.66:52691 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 06:57:56 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:56 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:56.093468 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:56.088844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:54075)
0926 06:57:56.089044 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:56.089056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:56.089147 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:56.089331 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:56.089332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:56.089511 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:56.089560 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:56.089561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.090225 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.090225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:56.090757 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:56.090758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.091076 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.091076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:56.091115 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:56.091566 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:56.091570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:56.093373 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
W20260926 06:57:56.093518 10461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:56.088926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:54075 (local address 127.1.74.126:38701)
0926 06:57:56.088966 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:56.088968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:56.089154 (+ 186us) server_negotiation.cc:407] Connection header received
0926 06:57:56.089215 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:56.089216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:56.089232 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:56.089285 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:56.089630 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.090152 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.090828 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.091010 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.091062 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:56.093451 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:54075: BlockingRecv error: recv got EOF from 127.1.74.67:54075 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 06:57:56 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:56 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:56.822340 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:56.813797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:58881)
0926 06:57:56.813978 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:56.813990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:56.814045 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:56.814229 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:56.814230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:56.814408 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:56.814486 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:56.814489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.815251 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.815275 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:56.815797 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:56.815798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.816128 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.816129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:56.816167 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:56.819942 (+ 3775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:56.819947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:56.822271 (+ 2324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":134,"threads_started":1}
W20260926 06:57:56.822479 10465 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:56.813920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:58881 (local address 127.1.74.126:38701)
0926 06:57:56.814037 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:56.814039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:56.814094 (+ 55us) server_negotiation.cc:407] Connection header received
0926 06:57:56.814125 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:56.814126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:56.814138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:56.814196 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:56.814586 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.815175 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.815872 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.816060 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.816115 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:56.822363 (+ 6248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:58881: BlockingRecv error: recv got EOF from 127.1.74.65:58881 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
Sep 26 06:57:56 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:56 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:56.973657 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:56.968780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:38575)
0926 06:57:56.968956 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:56.968970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:56.969021 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:56.969170 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:56.969171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:56.969335 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:56.969382 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:56.969384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.970181 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.970182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:56.970742 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:56.970744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.971181 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.971182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:56.971246 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:56.971723 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:56.971728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:56.973578 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
W20260926 06:57:56.973766 10465 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:56.968922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:38575 (local address 127.1.74.126:38701)
0926 06:57:56.968994 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:56.968996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:56.969021 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:57:56.969083 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:56.969084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:56.969097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:56.969145 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:56.969478 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.970093 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.970835 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:56.971076 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:56.971176 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:56.973707 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:38575: BlockingRecv error: recv got EOF from 127.1.74.66:38575 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 06:57:57 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:57 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:57.098795 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:57.093976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:49989)
0926 06:57:57.094178 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:57.094190 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:57.094282 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:57.094476 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:57.094477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:57.094655 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:57.094700 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:57.094702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.095440 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.095442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:57.095979 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:57.095980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.096370 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.096371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:57.096428 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:57.096860 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:57.096865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:57.098699 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":106,"threads_started":1}
W20260926 06:57:57.098845 10465 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:57.094096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:49989 (local address 127.1.74.126:38701)
0926 06:57:57.094176 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:57.094178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:57.094262 (+ 84us) server_negotiation.cc:407] Connection header received
0926 06:57:57.094360 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:57.094361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:57.094375 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:57.094467 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:57.094786 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.095348 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.096073 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.096282 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.096369 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:57.098779 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:49989: BlockingRecv error: recv got EOF from 127.1.74.67:49989 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 06:57:57 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:57 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:57.827767 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:57.822762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:42865)
0926 06:57:57.822931 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:57.822945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:57.823024 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:57.823443 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:57.823444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:57.823609 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:57.823654 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:57.823656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.824483 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.824484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:57.824989 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:57.824990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.825330 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.825331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:57.825369 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:57.825882 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:57.825887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:57.827710 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
W20260926 06:57:57.827916 10469 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:57.822840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:42865 (local address 127.1.74.126:38701)
0926 06:57:57.823198 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:57.823200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:57.823211 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:57:57.823235 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:57.823236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:57.823317 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:57.823398 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:57.823757 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.824382 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.825077 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.825264 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.825321 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:57.827824 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:42865: BlockingRecv error: recv got EOF from 127.1.74.65:42865 (error 108)
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":105,"threads_started":1}
Sep 26 06:57:57 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:57 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:57.978808 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:57.974276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:55301)
0926 06:57:57.974432 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:57.974444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:57.974510 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:57.974647 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:57.974648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:57.974800 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:57.974848 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:57.974850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.975557 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.975558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:57.976102 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:57.976104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.976461 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.976463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:57.976524 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:57.977012 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:57.977018 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:57.978751 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
W20260926 06:57:57.978940 10469 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:57.974347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:55301 (local address 127.1.74.126:38701)
0926 06:57:57.974417 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:57.974418 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:57.974507 (+ 89us) server_negotiation.cc:407] Connection header received
0926 06:57:57.974573 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:57.974574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:57.974584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:57.974627 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:57.974921 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.975487 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.976173 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:57.976352 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:57.976404 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:57.978859 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:55301: BlockingRecv error: recv got EOF from 127.1.74.66:55301 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 06:57:58 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:58 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:58.104411 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:58.099406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:48745)
0926 06:57:58.099556 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:58.099568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:58.099632 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:58.099777 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:58.099778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:58.100052 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:58.100129 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:58.100132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.100903 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.100904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:58.101487 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:58.101488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.101835 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.101835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:58.101876 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:58.102430 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:58.102435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:58.104316 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
W20260926 06:57:58.104451 10469 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:58.099478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:48745 (local address 127.1.74.126:38701)
0926 06:57:58.099560 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:58.099562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:58.099618 (+ 56us) server_negotiation.cc:407] Connection header received
0926 06:57:58.099677 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:58.099677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:58.099689 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:58.099759 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:58.100209 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.100833 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.101594 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.101771 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.101824 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:58.104390 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:48745: BlockingRecv error: recv got EOF from 127.1.74.67:48745 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 06:57:58 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:58 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:58.832617 10472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:58.828184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:54781)
0926 06:57:58.828329 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:58.828340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:58.828390 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:58.828509 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:58.828510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:58.828672 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:58.828718 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:58.828720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.829326 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.829327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:58.829842 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:58.829843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.830155 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.830155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:58.830192 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:58.830761 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:58.830765 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:58.832560 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260926 06:57:58.832772 10473 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:58.828263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:54781 (local address 127.1.74.126:38701)
0926 06:57:58.828366 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:58.828368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:58.828377 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:57:58.828445 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:58.828446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:58.828455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:58.828499 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:58.828780 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.829261 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.829913 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.830093 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.830143 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:58.832683 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:54781: BlockingRecv error: recv got EOF from 127.1.74.65:54781 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":48,"threads_started":1}
Sep 26 06:57:58 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:58 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:58.983704 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:58.979312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:43467)
0926 06:57:58.979459 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:58.979468 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:58.979524 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:58.979651 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:58.979652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:58.979819 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:58.979867 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:58.979868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.980458 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.980459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:58.980964 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:58.980965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.981292 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.981293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:58.981329 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:58.981828 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:58.981833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:58.983647 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
W20260926 06:57:58.983831 10473 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:58.979347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:43467 (local address 127.1.74.126:38701)
0926 06:57:58.979407 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:58.979414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:58.979522 (+ 108us) server_negotiation.cc:407] Connection header received
0926 06:57:58.979571 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:58.979572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:58.979582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:58.979645 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:58.979918 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.980386 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.981044 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:58.981228 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:58.981282 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:58.983730 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:43467: BlockingRecv error: recv got EOF from 127.1.74.66:43467 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 06:57:59 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:59 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:59.109289 10475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:59.104881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:50081)
0926 06:57:59.105042 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:59.105060 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:59.105108 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:59.105232 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:59.105233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:59.105390 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:59.105439 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:59.105440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.106046 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.106047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:59.106575 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:59.106576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.106880 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.106881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:59.106931 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:59.107463 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:59.107468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:59.109231 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
W20260926 06:57:59.109390 10473 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:59.104990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:50081 (local address 127.1.74.126:38701)
0926 06:57:59.105091 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:59.105097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:59.105107 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:57:59.105156 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:59.105156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:59.105169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:59.105211 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:59.105504 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.105980 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.106650 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.106820 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.106868 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:59.109318 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:50081: BlockingRecv error: recv got EOF from 127.1.74.67:50081 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 06:57:59 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:59 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:59.837679 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:59.833074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:39427)
0926 06:57:59.833203 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:59.833215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:59.833286 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:59.833464 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:59.833465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:59.833648 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:59.833692 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:59.833694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.834460 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.834461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:59.834988 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:59.834989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.835445 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.835447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:59.835511 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:59.835918 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:59.835922 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:59.837588 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
W20260926 06:57:59.837776 10477 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:59.833137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:39427 (local address 127.1.74.126:38701)
0926 06:57:59.833241 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:59.833243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:59.833271 (+ 28us) server_negotiation.cc:407] Connection header received
0926 06:57:59.833333 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:59.833334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:59.833345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:59.833431 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:59.833776 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.834375 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.835085 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.835366 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.835436 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:59.837707 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:39427: BlockingRecv error: recv got EOF from 127.1.74.65:39427 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":58,"threads_started":1}
Sep 26 06:57:59 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:57:59 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:57:59.988849 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:57:59.984181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:55577)
0926 06:57:59.984338 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:57:59.984348 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:57:59.984396 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:57:59.984516 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:57:59.984517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:57:59.984682 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:57:59.984727 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:59.984728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.985386 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.985387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:59.985924 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:57:59.985926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.986243 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.986243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:57:59.986284 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:59.986695 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:57:59.986700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:57:59.988729 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
W20260926 06:57:59.988878 10477 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:57:59.984302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:55577 (local address 127.1.74.126:38701)
0926 06:57:59.984364 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:57:59.984371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:57:59.984385 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:57:59.984446 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:57:59.984447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:57:59.984457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:57:59.984512 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:57:59.984812 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.985311 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.985995 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:57:59.986175 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:57:59.986236 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:57:59.988812 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:55577: BlockingRecv error: recv got EOF from 127.1.74.66:55577 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 06:58:00 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:00 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:58:00.114897 10477 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:00.109934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:41517 (local address 127.1.74.126:38701)
0926 06:58:00.109980 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:00.109983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:00.110107 (+ 124us) server_negotiation.cc:407] Connection header received
0926 06:58:00.110190 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:00.110190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:00.110204 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:00.110289 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:00.110611 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.111104 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.111864 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.112060 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.112144 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:00.114848 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:41517: BlockingRecv error: recv got EOF from 127.1.74.67:41517 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:58:00.114900 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:00.109883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:41517)
0926 06:58:00.110021 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:00.110033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:00.110123 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:00.110296 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:00.110297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:00.110484 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:00.110528 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:00.110530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.111190 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.111192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:00.111771 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:00.111773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.112148 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.112149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:00.112212 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:00.112699 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:00.112705 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:00.114768 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
Sep 26 06:58:00 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:00 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:00.842640 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:00.838188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:32867)
0926 06:58:00.838303 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:00.838314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:00.838381 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:00.838557 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:00.838558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:00.838769 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:00.838841 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:00.838843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.839496 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.839497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:00.840067 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:00.840069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.840411 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.840412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:00.840450 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:00.840961 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:00.840966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:00.842571 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
W20260926 06:58:00.842777 10481 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:00.838262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:32867 (local address 127.1.74.126:38701)
0926 06:58:00.838405 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:00.838407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:00.838416 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:58:00.838441 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:00.838442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:00.838453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:00.838539 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:00.838912 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.839433 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.840149 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.840345 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.840401 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:00.842661 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:32867: BlockingRecv error: recv got EOF from 127.1.74.65:32867 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
Sep 26 06:58:00 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:00 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:00.993953 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:00.989343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:39125)
0926 06:58:00.989488 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:00.989507 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:00.989569 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:00.989691 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:00.989692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:00.989850 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:00.989894 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:00.989896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.990710 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.990711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:00.991327 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:00.991330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.991704 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.991706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:00.991768 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:00.992180 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:00.992185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:00.993886 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
W20260926 06:58:00.994048 10481 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:00.989508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:39125 (local address 127.1.74.126:38701)
0926 06:58:00.989573 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:00.989574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:00.989585 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:58:00.989613 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:00.989614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:00.989624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:00.989672 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:00.989957 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.990643 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.991422 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:00.991621 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:00.991705 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:00.993980 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:39125: BlockingRecv error: recv got EOF from 127.1.74.66:39125 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 06:58:01 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:01 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:01.119875 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:01.115385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:51643)
0926 06:58:01.115523 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:01.115534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:01.115592 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:01.115706 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:01.115707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:01.115878 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:01.115923 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:01.115925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.116538 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.116538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:01.117061 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:01.117063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.117371 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.117372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:01.117409 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:01.117939 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:01.117943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:01.119811 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
W20260926 06:58:01.119968 10481 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:01.115469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:51643 (local address 127.1.74.126:38701)
0926 06:58:01.115535 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:01.115537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:01.115588 (+ 51us) server_negotiation.cc:407] Connection header received
0926 06:58:01.115633 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:01.115633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:01.115645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:01.115691 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:01.115979 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.116475 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.117128 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.117310 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.117358 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:01.119900 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:51643: BlockingRecv error: recv got EOF from 127.1.74.67:51643 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 06:58:01 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:01 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:01.851164 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:01.843120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:55911)
0926 06:58:01.843317 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:01.843333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:01.843388 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:01.843530 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:01.843531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:01.843692 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:01.843737 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:01.843739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.844381 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.844382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:01.844919 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:01.844920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.845249 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.845250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:01.845290 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:01.849278 (+ 3988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:01.849284 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:01.851104 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
W20260926 06:58:01.851352 10485 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:01.843215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:55911 (local address 127.1.74.126:38701)
0926 06:58:01.843386 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:01.843388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:01.843398 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:58:01.843440 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:01.843440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:01.843464 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:01.843524 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:01.843800 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.844313 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.844996 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.845183 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.845234 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:01.851221 (+ 5987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:55911: BlockingRecv error: recv got EOF from 127.1.74.65:55911 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
Sep 26 06:58:01 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:01 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:01.999317 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:01.994548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:33053)
0926 06:58:01.994717 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:01.994731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:01.994787 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:01.994915 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:01.994916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:01.995083 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:01.995128 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:01.995129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.995834 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.995835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:01.996414 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:01.996416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.996760 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.996761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:01.996811 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:01.997278 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:01.997282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:01.999187 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
W20260926 06:58:01.999372 10485 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:01.994698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:33053 (local address 127.1.74.126:38701)
0926 06:58:01.994762 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:01.994764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:01.994789 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:58:01.994840 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:01.994841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:01.994851 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:01.994894 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:01.995191 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.995762 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.996511 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:01.996695 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:01.996749 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:01.999304 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:33053: BlockingRecv error: recv got EOF from 127.1.74.66:33053 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 06:58:02 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:02 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:02.125689 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:02.120358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:39201)
0926 06:58:02.120573 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:02.120585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:02.120683 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:02.120842 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:02.120844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:02.121097 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:02.121142 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:02.121144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:02.121845 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:02.121846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:02.122501 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:02.122504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:02.122888 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:02.122889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:02.122950 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:02.123509 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:02.123515 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:02.125578 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
W20260926 06:58:02.125722 10485 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:02.120469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:39201 (local address 127.1.74.126:38701)
0926 06:58:02.120560 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:02.120562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:02.120663 (+ 101us) server_negotiation.cc:407] Connection header received
0926 06:58:02.120752 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:02.120753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:02.120768 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:02.120819 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:02.121221 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:02.121748 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:02.122593 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:02.122798 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:02.122886 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:02.125654 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:39201: BlockingRecv error: recv got EOF from 127.1.74.67:39201 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 06:58:02 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:02 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:02.856670 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:02.851822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:40789)
0926 06:58:02.851991 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:02.852006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:02.852074 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:02.852241 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:02.852242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:02.852403 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:02.852448 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:02.852449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:02.853116 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:02.853117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:02.853939 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:02.853942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:02.854295 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:02.854296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:02.854336 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:02.854765 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:02.854769 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:02.856560 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
W20260926 06:58:02.856699 10489 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:02.851945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:40789 (local address 127.1.74.126:38701)
0926 06:58:02.852117 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:02.852119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:02.852127 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:58:02.852150 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:02.852151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:02.852160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:02.852220 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:02.852519 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:02.853007 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:02.854031 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:02.854231 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:02.854286 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:02.856628 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:40789: BlockingRecv error: recv got EOF from 127.1.74.65:40789 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
Sep 26 06:58:03 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:03 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:03.004451 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:02.999779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:45329)
0926 06:58:02.999901 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:02.999915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:02.999983 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:03.000127 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:03.000128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:03.000310 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:03.000353 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:03.000354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.001120 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.001121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:03.001702 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:03.001704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.002057 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.002058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:03.002099 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:03.002552 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:03.002557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:03.004390 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
W20260926 06:58:03.004611 10489 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:02.999886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:45329 (local address 127.1.74.126:38701)
0926 06:58:02.999935 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:02.999937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:02.999963 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:58:03.000040 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:03.000041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:03.000053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:03.000122 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:03.000441 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.001020 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.001779 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.001977 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.002048 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:03.004517 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:45329: BlockingRecv error: recv got EOF from 127.1.74.66:45329 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 06:58:03 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:03 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.67@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:03.131008 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:03.126263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.67:37565)
0926 06:58:03.126414 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:03.126426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:03.126483 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:03.126663 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:03.126664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:03.126835 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:03.126880 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:03.126882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.127548 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.127549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:03.128072 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:03.128073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.128379 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.128380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:03.128418 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:03.128881 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:03.128885 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:03.130895 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
W20260926 06:58:03.131031 10489 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:03.126381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:37565 (local address 127.1.74.126:38701)
0926 06:58:03.126454 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:03.126456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:03.126469 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:58:03.126552 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:03.126552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:03.126566 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:03.126628 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:03.126949 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.127480 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.128145 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.128319 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.128368 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:03.130967 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.67:37565: BlockingRecv error: recv got EOF from 127.1.74.67:37565 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 06:58:03 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:03 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.65@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:03.862753 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:03.857260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.65:47379)
0926 06:58:03.857415 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:03.857430 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:03.857498 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:03.857674 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:03.857674 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:03.857843 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:03.857893 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:03.857895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.858637 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.858638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:03.859435 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:03.859443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.859817 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.859818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:03.859864 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:03.860504 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:03.860509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:03.862654 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":55,"threads_started":1}
W20260926 06:58:03.862807 10493 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:03.857429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:47379 (local address 127.1.74.126:38701)
0926 06:58:03.857537 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:03.857539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:03.857548 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:58:03.857582 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:03.857583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:03.857592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:03.857651 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:03.858041 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.858527 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.859516 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:03.859748 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:03.859802 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:03.862736 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.65:47379: BlockingRecv error: recv got EOF from 127.1.74.65:47379 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
Sep 26 06:58:04 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:58:04 dist-test-slave-sfvt krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.74.66@KRBTEST.COM for oryx/127.1.74.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:58:04.009289 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.004954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:38701 (local address 127.1.74.66:35215)
0926 06:58:04.005095 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:04.005111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:04.005206 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:04.005341 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:04.005342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:04.005502 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:04.005569 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:04.005571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.006211 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.006212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:04.006725 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:04.006727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.007033 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.007033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:04.007070 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:04.007532 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:04.007536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:04.009231 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.74.126:38701: Server oryx/127.1.74.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
W20260926 06:58:04.009433 10493 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:04.004998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:35215 (local address 127.1.74.126:38701)
0926 06:58:04.005060 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:04.005063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:04.005197 (+ 134us) server_negotiation.cc:407] Connection header received
0926 06:58:04.005262 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:04.005262 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:04.005272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:04.005324 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:04.005642 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.006142 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.006793 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.006971 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.007020 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:04.009350 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.74.66:35215: BlockingRecv error: recv got EOF from 127.1.74.66:35215 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:58:04.054008 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 10018
I20260926 06:58:04.058691 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 10170
I20260926 06:58:04.063030 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 10302
I20260926 06:58:04.066920 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 9948
2026-09-26T06:58:04Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17482 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T06:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:58:04Z Disabled control of system clock
I20260926 06:58:04.093886 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:35285
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:35285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 06:58:04.174361 10498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:04.174567 10498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:04.174587 10498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:04.175714 10498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:58:04.175750 10498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:04.175763 10498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:04.175776 10498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:58:04.175788 10498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:58:04.177058 10498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:35285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:35285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:04.177248 10498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:04.177496 10498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:04.183087 10504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:04.183197 10506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:04.183125 10498 server_base.cc:1034] running on GCE node
W20260926 06:58:04.183107 10503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:04.183570 10498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:04.183898 10498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:04.185045 10498 hybrid_clock.cc:648] HybridClock initialized: now 1790405884185017 us; error 55 us; skew 500 ppm
I20260926 06:58:04.186257 10498 webserver.cc:460] Webserver started at http://127.1.74.126:45815/ using document root <none> and password file <none>
I20260926 06:58:04.186504 10498 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:04.186550 10498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:04.186720 10498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:04.187884 10498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "6054263874344a1faed749a08002abca"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "e4533d1892d1fa0ae29fc279129805a4"
server_key_iv: "fa29d346bbf9b8dd226efc8e635f49d4"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.188253 10498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "6054263874344a1faed749a08002abca"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "e4533d1892d1fa0ae29fc279129805a4"
server_key_iv: "fa29d346bbf9b8dd226efc8e635f49d4"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.189545 10498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:58:04.190321 10512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.190506 10498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:58:04.190595 10498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "6054263874344a1faed749a08002abca"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "e4533d1892d1fa0ae29fc279129805a4"
server_key_iv: "fa29d346bbf9b8dd226efc8e635f49d4"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.190678 10498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:04.217011 10498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:04.217388 10498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:04.217576 10498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:04.221462 10498 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:35285
I20260926 06:58:04.221495 10564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:35285 every 8 connection(s)
I20260926 06:58:04.221850 10498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:58:04.222522 10565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:04.225224 10565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca: Bootstrap starting.
I20260926 06:58:04.225823 10565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:04.226135 10565 log.cc:824] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:04.226807 10565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca: No bootstrap required, opened a new log
I20260926 06:58:04.228421 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 10498
I20260926 06:58:04.228502 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:58:04.228672 1321 external_mini_cluster.cc:1428] Setting key ce791732b8fbd020c8b5e85338b22f8e
I20260926 06:58:04.228576 10565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6054263874344a1faed749a08002abca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35285 } }
I20260926 06:58:04.228744 10565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:04.228792 10565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6054263874344a1faed749a08002abca, State: Initialized, Role: FOLLOWER
I20260926 06:58:04.228947 10565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6054263874344a1faed749a08002abca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35285 } }
I20260926 06:58:04.229161 10565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:58:04.229223 10565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:58:04.229275 10565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:58:04.229995 10565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6054263874344a1faed749a08002abca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35285 } }
I20260926 06:58:04.230113 10565 leader_election.cc:302] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6054263874344a1faed749a08002abca; no voters:
I20260926 06:58:04.230281 10565 leader_election.cc:288] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:58:04.230363 10570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:58:04.230569 10565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:58:04.230571 10570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [term 1 LEADER]: Becoming Leader. State: Replica: 6054263874344a1faed749a08002abca, State: Running, Role: LEADER
I20260926 06:58:04.230818 10570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6054263874344a1faed749a08002abca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35285 } }
I20260926 06:58:04.231213 10572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6054263874344a1faed749a08002abca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6054263874344a1faed749a08002abca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35285 } } }
I20260926 06:58:04.231222 10571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6054263874344a1faed749a08002abca. Latest consensus state: current_term: 1 leader_uuid: "6054263874344a1faed749a08002abca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6054263874344a1faed749a08002abca" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 35285 } } }
I20260926 06:58:04.231413 10572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [sys.catalog]: This master's current role is: LEADER
I20260926 06:58:04.231421 10571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca [sys.catalog]: This master's current role is: LEADER
I20260926 06:58:04.231688 10579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:58:04.232465 10579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:58:04.234157 10579 catalog_manager.cc:1347] Generated new cluster ID: a239ef5ddb4244048c5dd0ce6bc49e6f
I20260926 06:58:04.234215 10579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:58:04.235631 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.229183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.1.74.126:35285)
0926 06:58:04.229432 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:04.229442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:04.229475 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:58:04.229596 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:04.229603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:04.229692 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:04.229850 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:04.230152 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.232023 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.233147 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.233768 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.233895 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:04.234857 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:04.234862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:04.234862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:58:04.235084 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:04.235126 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:58:04.235140 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:04.235211 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:04.235344 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 06:58:04.235422 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260926 06:58:04.241633 10579 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:58:04.242185 10579 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:58:04.245718 10579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6054263874344a1faed749a08002abca: Generated new TSK 0
I20260926 06:58:04.245914 10579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:58:04.250065 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35285
--builtin_ntp_servers=127.1.74.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 06:58:04.333081 10589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:04.333298 10589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:04.333323 10589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:04.334590 10589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:04.334633 10589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:04.334663 10589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:58:04.336557 10589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:04.336807 10589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:04.337153 10589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:04.343201 10597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:04.343333 10589 server_base.cc:1034] running on GCE node
W20260926 06:58:04.343178 10594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:04.343178 10595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:04.343689 10589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:04.343966 10589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:04.345106 10589 hybrid_clock.cc:648] HybridClock initialized: now 1790405884345086 us; error 34 us; skew 500 ppm
I20260926 06:58:04.346426 10589 webserver.cc:460] Webserver started at http://127.1.74.65:37943/ using document root <none> and password file <none>
I20260926 06:58:04.346686 10589 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:04.346732 10589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:04.346892 10589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:04.348022 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "1e542ccb4aa249e489bdebb94bf69025"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "70cd8b8dcd07545813f34ac082719a63"
server_key_iv: "67f27c36567de908703e77c7295de865"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.348392 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "1e542ccb4aa249e489bdebb94bf69025"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "70cd8b8dcd07545813f34ac082719a63"
server_key_iv: "67f27c36567de908703e77c7295de865"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.349735 10589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:58:04.350479 10603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.350648 10589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:58:04.350720 10589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "1e542ccb4aa249e489bdebb94bf69025"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "70cd8b8dcd07545813f34ac082719a63"
server_key_iv: "67f27c36567de908703e77c7295de865"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.350791 10589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:04.362710 10589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:04.363014 10589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:04.363163 10589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:04.363425 10589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:04.363786 10589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:04.363842 10589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.363883 10589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:04.363909 10589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.369333 10589 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:40001
I20260926 06:58:04.369411 10716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:40001 every 8 connection(s)
I20260926 06:58:04.369758 10589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:58:04.374680 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.370503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:45845 (local address 127.1.74.126:35285)
0926 06:58:04.370569 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:04.370573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:04.370823 (+ 250us) server_negotiation.cc:407] Connection header received
0926 06:58:04.371344 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:04.371345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:04.371357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:04.371406 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:04.372025 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.372514 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.373459 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.373655 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.373708 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:04.374438 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:04.374439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:04.374439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:58:04.374444 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:04.374457 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:58:04.374460 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:04.374500 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:04.374580 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 06:58:04.374607 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:58:04.374855 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.370504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:35285 (local address 127.1.74.65:45845)
0926 06:58:04.370703 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:04.370738 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:04.371237 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:04.371468 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:04.371471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:04.371479 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:04.371958 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:04.371962 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.372599 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.372600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:04.373385 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:04.373387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.373732 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.373732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:04.373839 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:04.374313 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:58:04.374314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:04.374341 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:58:04.374363 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:04.374514 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:04.374517 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:04.374519 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:58:04.374570 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:58:04.374645 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260926 06:58:04.375180 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 10589
I20260926 06:58:04.375294 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:58:04.375355 10717 heartbeater.cc:347] Connected to a master server at 127.1.74.126:35285
I20260926 06:58:04.375445 10717 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:04.375504 1321 external_mini_cluster.cc:1428] Setting key 5ae7a1a7e72d7e7239d960eaa85bb049
I20260926 06:58:04.375587 10717 heartbeater.cc:511] Master 127.1.74.126:35285 requested a full tablet report, sending...
I20260926 06:58:04.376255 10528 ts_manager.cc:194] Registered new tserver with Master: 1e542ccb4aa249e489bdebb94bf69025 (127.1.74.65:40001)
I20260926 06:58:04.376497 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35285
--builtin_ntp_servers=127.1.74.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 06:58:04.376959 10528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.65:45845
W20260926 06:58:04.461278 10720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:04.461476 10720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:04.461498 10720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:04.462565 10720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:04.462599 10720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:04.462623 10720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:58:04.463959 10720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:04.464145 10720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:04.464388 10720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:04.469214 10726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:04.469223 10725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:04.469230 10728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:04.469758 10720 server_base.cc:1034] running on GCE node
I20260926 06:58:04.469987 10720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:04.470247 10720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:04.471422 10720 hybrid_clock.cc:648] HybridClock initialized: now 1790405884471404 us; error 27 us; skew 500 ppm
I20260926 06:58:04.472609 10720 webserver.cc:460] Webserver started at http://127.1.74.66:33985/ using document root <none> and password file <none>
I20260926 06:58:04.472858 10720 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:04.472929 10720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:04.473062 10720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:04.474128 10720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "2f7ef102d8b54ef0b71b4ce316eeea11"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "16310d57059abb088ba7f9ac4b96c68f"
server_key_iv: "1af20e826c6fdde6447534acb5b9b794"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.474499 10720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "2f7ef102d8b54ef0b71b4ce316eeea11"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "16310d57059abb088ba7f9ac4b96c68f"
server_key_iv: "1af20e826c6fdde6447534acb5b9b794"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.475819 10720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:58:04.476517 10734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.476869 10720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:58:04.476966 10720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "2f7ef102d8b54ef0b71b4ce316eeea11"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "16310d57059abb088ba7f9ac4b96c68f"
server_key_iv: "1af20e826c6fdde6447534acb5b9b794"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.477039 10720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:04.505117 10720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:04.505451 10720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:04.505604 10720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:04.505841 10720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:04.506186 10720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:04.506239 10720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.506281 10720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:04.506310 10720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.511956 10720 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:34821
I20260926 06:58:04.512037 10847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:34821 every 8 connection(s)
I20260926 06:58:04.512365 10720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:58:04.517304 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.513099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:56625 (local address 127.1.74.126:35285)
0926 06:58:04.513149 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:04.513151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:04.513377 (+ 226us) server_negotiation.cc:407] Connection header received
0926 06:58:04.513873 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:04.513875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:04.513886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:04.513939 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:04.514571 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.515066 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.516107 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.516309 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.516363 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:04.517083 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:04.517084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:04.517085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:58:04.517089 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:04.517101 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:58:04.517104 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:04.517166 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:04.517226 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 06:58:04.517246 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:58:04.517518 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.513073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:35285 (local address 127.1.74.66:56625)
0926 06:58:04.513278 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:04.513310 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:04.513782 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:04.513992 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:04.513995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:04.514003 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:04.514501 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:04.514506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.515151 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.515151 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:04.516034 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:04.516036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.516389 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.516390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:04.516496 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:04.516964 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:58:04.516964 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:04.516991 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:58:04.517013 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:04.517173 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:04.517176 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:04.517178 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:58:04.517228 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:58:04.517308 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260926 06:58:04.518033 10848 heartbeater.cc:347] Connected to a master server at 127.1.74.126:35285
I20260926 06:58:04.518133 10848 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:04.518321 10848 heartbeater.cc:511] Master 127.1.74.126:35285 requested a full tablet report, sending...
I20260926 06:58:04.518711 10528 ts_manager.cc:194] Registered new tserver with Master: 2f7ef102d8b54ef0b71b4ce316eeea11 (127.1.74.66:34821)
I20260926 06:58:04.519069 10528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.66:56625
I20260926 06:58:04.521350 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 10720
I20260926 06:58:04.521414 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:58:04.521596 1321 external_mini_cluster.cc:1428] Setting key 3c1b277d2fb09122a18dd38661bceca5
I20260926 06:58:04.522550 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35285
--builtin_ntp_servers=127.1.74.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 06:58:04.602772 10851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:04.602969 10851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:04.602989 10851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:04.604212 10851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:04.604255 10851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:04.604281 10851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:58:04.605729 10851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:04.605973 10851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:04.606280 10851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:04.611685 10856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:04.611706 10859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:04.611716 10857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:04.611862 10851 server_base.cc:1034] running on GCE node
I20260926 06:58:04.612074 10851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:04.612329 10851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:04.613493 10851 hybrid_clock.cc:648] HybridClock initialized: now 1790405884613476 us; error 26 us; skew 500 ppm
I20260926 06:58:04.614722 10851 webserver.cc:460] Webserver started at http://127.1.74.67:34833/ using document root <none> and password file <none>
I20260926 06:58:04.614959 10851 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:04.615028 10851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:04.615178 10851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:04.616298 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "d511baa776d04695a8063a1c728d5c45"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "41b53eb46206285c2f8cece655b1a34c"
server_key_iv: "0ceaa28f3766c0a6df8802504ea7552a"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.616670 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "d511baa776d04695a8063a1c728d5c45"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "41b53eb46206285c2f8cece655b1a34c"
server_key_iv: "0ceaa28f3766c0a6df8802504ea7552a"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.617949 10851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:58:04.618620 10865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.618772 10851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:58:04.618866 10851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "d511baa776d04695a8063a1c728d5c45"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "41b53eb46206285c2f8cece655b1a34c"
server_key_iv: "0ceaa28f3766c0a6df8802504ea7552a"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.618944 10851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:04.637840 10851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:04.638163 10851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:04.638325 10851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:04.638556 10851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:04.638882 10851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:04.638938 10851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.638981 10851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:04.639007 10851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.644553 10851 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:38369
I20260926 06:58:04.644630 10978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:38369 every 8 connection(s)
I20260926 06:58:04.644961 10851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:58:04.647078 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 10851
I20260926 06:58:04.647303 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:58:04.647486 1321 external_mini_cluster.cc:1428] Setting key 6b9f149e482c027605a6c6cc7f9b8966
I20260926 06:58:04.650457 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.645684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:40447 (local address 127.1.74.126:35285)
0926 06:58:04.645728 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:04.645730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:04.646003 (+ 273us) server_negotiation.cc:407] Connection header received
0926 06:58:04.646456 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:04.646457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:04.646467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:04.646519 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:04.647279 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.647984 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.649034 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.649253 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.649342 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:04.650179 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:04.650181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:04.650181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:58:04.650185 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:04.650199 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:58:04.650204 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:04.650259 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:04.650351 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 06:58:04.650379 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:58:04.650681 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.645701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:35285 (local address 127.1.74.67:40447)
0926 06:58:04.645872 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:04.645905 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:04.646373 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:04.646572 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:04.646576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:04.646583 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:04.647069 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:04.647074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.648076 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.648077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:04.648938 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:04.648940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:04.649351 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:04.649352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:04.649479 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:04.650037 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:58:04.650037 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:04.650065 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:58:04.650086 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:04.650261 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:04.650265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:04.650268 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:58:04.650355 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 06:58:04.650449 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260926 06:58:04.651105 10979 heartbeater.cc:347] Connected to a master server at 127.1.74.126:35285
I20260926 06:58:04.651206 10979 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:04.651408 10979 heartbeater.cc:511] Master 127.1.74.126:35285 requested a full tablet report, sending...
I20260926 06:58:04.651762 10528 ts_manager.cc:194] Registered new tserver with Master: d511baa776d04695a8063a1c728d5c45 (127.1.74.67:38369)
I20260926 06:58:04.652068 10528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.67:40447
I20260926 06:58:04.659394 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 06:58:04.661257 10569 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:04.660781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.1.74.126:35285)
0926 06:58:04.660862 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:04.660864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:04.660935 (+ 71us) server_negotiation.cc:407] Connection header received
0926 06:58:04.660975 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:04.660975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:04.660986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:04.661036 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:04.661210 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35024: BlockingRecv error: recv got EOF from 127.0.0.1:35024 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 06:58:04.661310 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 10589
I20260926 06:58:04.666407 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 10720
I20260926 06:58:04.670328 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 10851
I20260926 06:58:04.674216 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 10498
2026-09-26T06:58:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (606 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T06:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:58:04Z Disabled control of system clock
I20260926 06:58:04.700368 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33563
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:34741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:33563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:58:04.780345 10990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:04.780545 10990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:04.780566 10990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:04.781649 10990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:58:04.781683 10990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:04.781697 10990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:04.781708 10990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:58:04.781720 10990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:58:04.782994 10990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:33563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:04.783183 10990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:04.783499 10990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:04.788220 10995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:04.788262 10996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:04.788277 10998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:04.788516 10990 server_base.cc:1034] running on GCE node
I20260926 06:58:04.788703 10990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:04.788942 10990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:04.790078 10990 hybrid_clock.cc:648] HybridClock initialized: now 1790405884790070 us; error 30 us; skew 500 ppm
I20260926 06:58:04.791218 10990 webserver.cc:460] Webserver started at http://127.1.74.126:42617/ using document root <none> and password file <none>
I20260926 06:58:04.791471 10990 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:04.791518 10990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:04.791606 10990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:04.792613 10990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "6d929c580c2744ccb2ee51491f954c3b"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "effe8da68bf92a613bd36ff032920ff0"
server_key_iv: "3c9626cef19e7f06aac72450c0b08aaf"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.792908 10990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "6d929c580c2744ccb2ee51491f954c3b"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "effe8da68bf92a613bd36ff032920ff0"
server_key_iv: "3c9626cef19e7f06aac72450c0b08aaf"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.794210 10990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:58:04.794850 11004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.795022 10990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:58:04.795089 10990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "6d929c580c2744ccb2ee51491f954c3b"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "effe8da68bf92a613bd36ff032920ff0"
server_key_iv: "3c9626cef19e7f06aac72450c0b08aaf"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.795138 10990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:04.798470 10990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:04.798695 10990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:04.798789 10990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:04.802536 10990 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:33563
I20260926 06:58:04.802562 11056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:33563 every 8 connection(s)
I20260926 06:58:04.802881 10990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:58:04.803566 11057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:04.804105 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 10990
I20260926 06:58:04.804205 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:58:04.804400 1321 external_mini_cluster.cc:1428] Setting key c5d4a78ca1d3004b11f945da18b825da
I20260926 06:58:04.807220 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.805101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.1.74.126:33563)
0926 06:58:04.805303 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:04.805313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:04.805346 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:58:04.805828 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:04.805837 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:04.805893 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:04.806041 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:04.806730 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:04.806734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:04.806735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:58:04.806739 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:04.806785 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:58:04.806800 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:04.806881 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:04.806970 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 06:58:04.807029 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260926 06:58:04.807674 11057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b: Bootstrap starting.
I20260926 06:58:04.808313 11057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:04.808530 11057 log.cc:824] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:04.809135 11057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b: No bootstrap required, opened a new log
I20260926 06:58:04.810456 11057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d929c580c2744ccb2ee51491f954c3b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33563 } }
I20260926 06:58:04.810585 11057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:04.810606 11057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d929c580c2744ccb2ee51491f954c3b, State: Initialized, Role: FOLLOWER
I20260926 06:58:04.810734 11057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d929c580c2744ccb2ee51491f954c3b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33563 } }
I20260926 06:58:04.810822 11057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:58:04.810845 11057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:58:04.810873 11057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:58:04.811491 11057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d929c580c2744ccb2ee51491f954c3b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33563 } }
I20260926 06:58:04.811575 11057 leader_election.cc:302] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d929c580c2744ccb2ee51491f954c3b; no voters:
I20260926 06:58:04.811697 11057 leader_election.cc:288] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:58:04.811941 11057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:58:04.812026 11062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:58:04.812162 11062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [term 1 LEADER]: Becoming Leader. State: Replica: 6d929c580c2744ccb2ee51491f954c3b, State: Running, Role: LEADER
I20260926 06:58:04.812278 11062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d929c580c2744ccb2ee51491f954c3b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33563 } }
I20260926 06:58:04.812603 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d929c580c2744ccb2ee51491f954c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d929c580c2744ccb2ee51491f954c3b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33563 } } }
I20260926 06:58:04.812696 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [sys.catalog]: This master's current role is: LEADER
I20260926 06:58:04.812846 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d929c580c2744ccb2ee51491f954c3b. Latest consensus state: current_term: 1 leader_uuid: "6d929c580c2744ccb2ee51491f954c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d929c580c2744ccb2ee51491f954c3b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33563 } } }
I20260926 06:58:04.812904 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b [sys.catalog]: This master's current role is: LEADER
I20260926 06:58:04.813021 11071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:58:04.813508 11071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:58:04.814996 11071 catalog_manager.cc:1347] Generated new cluster ID: 0387ffe8fee7499484e711bbe6c1e48c
I20260926 06:58:04.815039 11071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:58:04.820290 11071 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:58:04.820451 11071 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:58:04.827224 11071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d929c580c2744ccb2ee51491f954c3b: Generated new TSK 0
I20260926 06:58:04.827446 11071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:58:04.834555 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33563
--builtin_ntp_servers=127.1.74.84:34741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:58:04.920457 11081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:04.920661 11081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:04.920684 11081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:04.921846 11081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:04.921885 11081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:04.921911 11081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:58:04.923241 11081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:04.923533 11081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:04.923871 11081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:04.929675 11086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:04.929700 11087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:04.929976 11081 server_base.cc:1034] running on GCE node
W20260926 06:58:04.930171 11089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:04.930399 11081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:04.930711 11081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:04.931869 11081 hybrid_clock.cc:648] HybridClock initialized: now 1790405884931856 us; error 29 us; skew 500 ppm
I20260926 06:58:04.933159 11081 webserver.cc:460] Webserver started at http://127.1.74.65:40169/ using document root <none> and password file <none>
I20260926 06:58:04.933421 11081 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:04.933494 11081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:04.933650 11081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:04.934721 11081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "d2aeaf14004243c8abee60bdda7bfcb4"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "9b7db35468c4e6bc4efb33c98e5bdf0b"
server_key_iv: "d5bc2ba1cb9072225a09f6b6f9026646"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.935096 11081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "d2aeaf14004243c8abee60bdda7bfcb4"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "9b7db35468c4e6bc4efb33c98e5bdf0b"
server_key_iv: "d5bc2ba1cb9072225a09f6b6f9026646"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.936499 11081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:58:04.937291 11095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.937475 11081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:58:04.937567 11081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "d2aeaf14004243c8abee60bdda7bfcb4"
format_stamp: "Formatted at 2026-09-26 06:58:04 on dist-test-slave-sfvt"
server_key: "9b7db35468c4e6bc4efb33c98e5bdf0b"
server_key_iv: "d5bc2ba1cb9072225a09f6b6f9026646"
server_key_version: "encryptionkey@0"
I20260926 06:58:04.937652 11081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:04.941111 11081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:04.941380 11081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:04.941519 11081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:04.941725 11081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:04.942031 11081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:04.942086 11081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.942126 11081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:04.942153 11081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:04.947870 11081 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:39745
I20260926 06:58:04.947902 11208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:39745 every 8 connection(s)
I20260926 06:58:04.948236 11081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:58:04.949491 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 11081
I20260926 06:58:04.949558 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:58:04.949757 1321 external_mini_cluster.cc:1428] Setting key b157997e42eecc9664d119e3a471f521
I20260926 06:58:04.950774 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33563
--builtin_ntp_servers=127.1.74.84:34741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 06:58:04.951542 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.948959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:57237 (local address 127.1.74.126:33563)
0926 06:58:04.949026 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:04.949029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:04.949280 (+ 251us) server_negotiation.cc:407] Connection header received
0926 06:58:04.949839 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:04.949841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:04.949847 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:04.949898 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:04.951246 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:04.951247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:04.951248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:58:04.951268 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:04.951286 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:58:04.951291 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:04.951361 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:04.951426 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 06:58:04.951440 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:58:04.951659 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:04.948958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33563 (local address 127.1.74.65:57237)
0926 06:58:04.949160 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:04.949192 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:04.949716 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:04.949968 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:04.949972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:04.949982 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:04.951102 (+ 1120us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:58:04.951102 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:04.951130 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:58:04.951157 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:04.951359 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:04.951362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:04.951364 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:58:04.951423 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:58:04.951472 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260926 06:58:04.952126 11209 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33563
I20260926 06:58:04.952214 11209 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:04.952379 11209 heartbeater.cc:511] Master 127.1.74.126:33563 requested a full tablet report, sending...
I20260926 06:58:04.952922 11021 ts_manager.cc:194] Registered new tserver with Master: d2aeaf14004243c8abee60bdda7bfcb4 (127.1.74.65:39745)
W20260926 06:58:05.031236 11212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:05.031455 11212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:05.031476 11212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:05.032560 11212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:05.032593 11212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:05.032618 11212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:58:05.033923 11212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:05.034356 11212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:05.034629 11212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:05.039920 11217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:05.039973 11218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:05.039920 11220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:05.040009 11212 server_base.cc:1034] running on GCE node
I20260926 06:58:05.040274 11212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:05.040483 11212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:05.041612 11212 hybrid_clock.cc:648] HybridClock initialized: now 1790405885041592 us; error 37 us; skew 500 ppm
I20260926 06:58:05.042873 11212 webserver.cc:460] Webserver started at http://127.1.74.66:36439/ using document root <none> and password file <none>
I20260926 06:58:05.043113 11212 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:05.043162 11212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:05.043344 11212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:05.044411 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "07414ff0deb24447a19e957b0ac7c9bf"
format_stamp: "Formatted at 2026-09-26 06:58:05 on dist-test-slave-sfvt"
server_key: "3f7380d858099072c66b94a1b870a788"
server_key_iv: "51a0149f6fa9e81c60c44ef1392c4d44"
server_key_version: "encryptionkey@0"
I20260926 06:58:05.044759 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "07414ff0deb24447a19e957b0ac7c9bf"
format_stamp: "Formatted at 2026-09-26 06:58:05 on dist-test-slave-sfvt"
server_key: "3f7380d858099072c66b94a1b870a788"
server_key_iv: "51a0149f6fa9e81c60c44ef1392c4d44"
server_key_version: "encryptionkey@0"
I20260926 06:58:05.045998 11212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:58:05.046650 11226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:05.046824 11212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:58:05.046923 11212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "07414ff0deb24447a19e957b0ac7c9bf"
format_stamp: "Formatted at 2026-09-26 06:58:05 on dist-test-slave-sfvt"
server_key: "3f7380d858099072c66b94a1b870a788"
server_key_iv: "51a0149f6fa9e81c60c44ef1392c4d44"
server_key_version: "encryptionkey@0"
I20260926 06:58:05.047001 11212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:05.050297 11212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:05.050557 11212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:05.050700 11212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:05.050904 11212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:05.051198 11212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:05.051254 11212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:05.051304 11212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:05.051333 11212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:05.056802 11212 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:37903
I20260926 06:58:05.056841 11339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:37903 every 8 connection(s)
I20260926 06:58:05.057163 11212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:58:05.059999 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:05.057865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:50031 (local address 127.1.74.126:33563)
0926 06:58:05.057939 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:05.057941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:05.058112 (+ 171us) server_negotiation.cc:407] Connection header received
0926 06:58:05.058547 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:05.058549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:05.058553 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:05.058605 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:05.059815 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:05.059816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:05.059817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:58:05.059821 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:05.059831 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:58:05.059835 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:05.059877 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:05.059935 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 06:58:05.059947 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:58:05.060175 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:05.057814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33563 (local address 127.1.74.66:50031)
0926 06:58:05.057999 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:05.058031 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:05.058468 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:05.058657 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:05.058660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:05.058668 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:05.059692 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:58:05.059693 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:05.059719 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:58:05.059745 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:05.059877 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:05.059879 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:05.059881 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:58:05.059931 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:58:05.059976 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260926 06:58:05.060776 11340 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33563
I20260926 06:58:05.060881 11340 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:05.061022 11340 heartbeater.cc:511] Master 127.1.74.126:33563 requested a full tablet report, sending...
I20260926 06:58:05.061369 11021 ts_manager.cc:194] Registered new tserver with Master: 07414ff0deb24447a19e957b0ac7c9bf (127.1.74.66:37903)
I20260926 06:58:05.065914 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 11212
I20260926 06:58:05.065995 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:58:05.066125 1321 external_mini_cluster.cc:1428] Setting key 1559aaf27223ba58ec41be8b925a8da2
I20260926 06:58:05.066929 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33563
--builtin_ntp_servers=127.1.74.84:34741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:58:05.146873 11343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:05.147112 11343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:05.147149 11343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:05.148310 11343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:05.148382 11343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:05.148424 11343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:58:05.149801 11343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:05.150060 11343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:05.150363 11343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:05.155355 11349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:05.155367 11348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:05.155367 11351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:05.155486 11343 server_base.cc:1034] running on GCE node
I20260926 06:58:05.155774 11343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:05.155978 11343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:05.157119 11343 hybrid_clock.cc:648] HybridClock initialized: now 1790405885157108 us; error 26 us; skew 500 ppm
I20260926 06:58:05.158349 11343 webserver.cc:460] Webserver started at http://127.1.74.67:44843/ using document root <none> and password file <none>
I20260926 06:58:05.158545 11343 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:05.158583 11343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:05.158669 11343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:05.159729 11343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "36a974245daa43c59b41706c26924287"
format_stamp: "Formatted at 2026-09-26 06:58:05 on dist-test-slave-sfvt"
server_key: "eeb828620dac683a9dd2576042f248e4"
server_key_iv: "331da403458f4cc22dc3b2772db61d8c"
server_key_version: "encryptionkey@0"
I20260926 06:58:05.160017 11343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "36a974245daa43c59b41706c26924287"
format_stamp: "Formatted at 2026-09-26 06:58:05 on dist-test-slave-sfvt"
server_key: "eeb828620dac683a9dd2576042f248e4"
server_key_iv: "331da403458f4cc22dc3b2772db61d8c"
server_key_version: "encryptionkey@0"
I20260926 06:58:05.161168 11343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:58:05.161861 11357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:05.162024 11343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:58:05.162102 11343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "36a974245daa43c59b41706c26924287"
format_stamp: "Formatted at 2026-09-26 06:58:05 on dist-test-slave-sfvt"
server_key: "eeb828620dac683a9dd2576042f248e4"
server_key_iv: "331da403458f4cc22dc3b2772db61d8c"
server_key_version: "encryptionkey@0"
I20260926 06:58:05.162158 11343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:05.165331 11343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:05.165526 11343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:05.165617 11343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:05.165759 11343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:05.165988 11343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:05.166018 11343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:05.166036 11343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:05.166049 11343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:05.171391 11343 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:36305
I20260926 06:58:05.171456 11470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:36305 every 8 connection(s)
I20260926 06:58:05.171667 11343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:58:05.174908 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:05.172310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:44313 (local address 127.1.74.126:33563)
0926 06:58:05.172357 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:05.172360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:05.172631 (+ 271us) server_negotiation.cc:407] Connection header received
0926 06:58:05.173264 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:05.173266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:05.173272 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:05.173349 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:05.174602 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:05.174604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:05.174604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:58:05.174609 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:05.174626 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:58:05.174630 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:05.174736 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:05.174816 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 06:58:05.174838 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:58:05.175093 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:05.172299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33563 (local address 127.1.74.67:44313)
0926 06:58:05.172499 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:05.172529 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:05.173144 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:05.173447 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:05.173451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:05.173462 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:05.174436 (+ 974us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:58:05.174436 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:05.174463 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:58:05.174489 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:05.174735 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:05.174741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:05.174744 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:58:05.174826 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:58:05.174885 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260926 06:58:05.175575 11471 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33563
I20260926 06:58:05.175649 11471 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:05.175819 11471 heartbeater.cc:511] Master 127.1.74.126:33563 requested a full tablet report, sending...
I20260926 06:58:05.176159 11021 ts_manager.cc:194] Registered new tserver with Master: 36a974245daa43c59b41706c26924287 (127.1.74.67:36305)
I20260926 06:58:05.181169 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 11343
I20260926 06:58:05.181248 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:58:05.181480 1321 external_mini_cluster.cc:1428] Setting key c492024827864210b7f87d4a68d862ce
I20260926 06:58:05.182181 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 06:58:05.183976 11060 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:58:05.183533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.1.74.126:33563)
0926 06:58:05.183593 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:05.183595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:05.183694 (+ 99us) server_negotiation.cc:407] Connection header received
0926 06:58:05.183718 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:05.183718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:05.183722 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:05.183771 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:05.183898 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56882: BlockingRecv error: recv got EOF from 127.0.0.1:56882 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:58:05.184083 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 11081
I20260926 06:58:05.188679 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 11212
I20260926 06:58:05.193009 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 11343
I20260926 06:58:05.197157 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 10990
2026-09-26T06:58:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (523 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:58:05 dist-test-slave-sfvt krb5kdc[11481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:58:05 dist-test-slave-sfvt krb5kdc[11481](info): set up 2 sockets
Sep 26 06:58:05 dist-test-slave-sfvt krb5kdc[11481](info): commencing operation
krb5kdc: starting...
W20260926 06:58:06.232240 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:58:06 dist-test-slave-sfvt krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:58:06.350021 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45113
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:45113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:58:06.429252 11494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:06.429536 11494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:06.429574 11494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:06.430720 11494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:58:06.430801 11494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:06.430824 11494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:06.430845 11494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:58:06.430871 11494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:58:06.432339 11494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:45113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:06.432641 11494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:06.432876 11494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:06.437786 11502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:06.437811 11499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:06.437819 11500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:06.437940 11494 server_base.cc:1034] running on GCE node
I20260926 06:58:06.438275 11494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:06.438570 11494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:06.439734 11494 hybrid_clock.cc:648] HybridClock initialized: now 1790405886439721 us; error 46 us; skew 500 ppm
Sep 26 06:58:06 dist-test-slave-sfvt krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:58:06.442128 11494 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:58:06.442592 11494 webserver.cc:460] Webserver started at http://127.1.74.126:43037/ using document root <none> and password file <none>
I20260926 06:58:06.442824 11494 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:06.442900 11494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:06.443049 11494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:06.444168 11494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "e344fc370f574129b6e3de5ce71d2507"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "0d5763dc5cc493caeefe4570261aef32"
server_key_iv: "3620ada826daaf98e35a949222593cb3"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.444533 11494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "e344fc370f574129b6e3de5ce71d2507"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "0d5763dc5cc493caeefe4570261aef32"
server_key_iv: "3620ada826daaf98e35a949222593cb3"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.445839 11494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:58:06.446544 11509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.446717 11494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:58:06.446813 11494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "e344fc370f574129b6e3de5ce71d2507"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "0d5763dc5cc493caeefe4570261aef32"
server_key_iv: "3620ada826daaf98e35a949222593cb3"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.446910 11494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:06.455987 11494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:06.457918 11494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:06.458087 11494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:06.462133 11494 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:45113
I20260926 06:58:06.462170 11561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:45113 every 8 connection(s)
I20260926 06:58:06.462525 11494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:58:06.463574 11562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:06.464274 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 11494
I20260926 06:58:06.464362 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:58:06.464548 1321 external_mini_cluster.cc:1428] Setting key 277d49f676eeb9e0c4d46f5a0c30c518
I20260926 06:58:06.466650 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507: Bootstrap starting.
I20260926 06:58:06.467233 11562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:06.467594 11562 log.cc:824] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:06.468269 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507: No bootstrap required, opened a new log
I20260926 06:58:06.470562 11562 raft_consensus.cc:348] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e344fc370f574129b6e3de5ce71d2507" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45113 } }
I20260926 06:58:06.470742 11562 raft_consensus.cc:374] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:06.470777 11562 raft_consensus.cc:729] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e344fc370f574129b6e3de5ce71d2507, State: Initialized, Role: FOLLOWER
I20260926 06:58:06.470912 11562 consensus_queue.cc:260] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e344fc370f574129b6e3de5ce71d2507" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45113 } }
I20260926 06:58:06.471009 11562 raft_consensus.cc:388] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:58:06.471041 11562 raft_consensus.cc:482] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:58:06.471087 11562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 06:58:06 dist-test-slave-sfvt krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:58:06.471786 11562 raft_consensus.cc:504] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e344fc370f574129b6e3de5ce71d2507" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45113 } }
I20260926 06:58:06.471902 11562 leader_election.cc:302] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e344fc370f574129b6e3de5ce71d2507; no voters:
I20260926 06:58:06.472071 11562 leader_election.cc:288] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:58:06.472106 11567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:58:06.472314 11567 raft_consensus.cc:686] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [term 1 LEADER]: Becoming Leader. State: Replica: e344fc370f574129b6e3de5ce71d2507, State: Running, Role: LEADER
I20260926 06:58:06.472468 11567 consensus_queue.cc:237] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e344fc370f574129b6e3de5ce71d2507" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45113 } }
I20260926 06:58:06.472352 11562 sys_catalog.cc:564] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:58:06.472898 11569 sys_catalog.cc:455] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e344fc370f574129b6e3de5ce71d2507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e344fc370f574129b6e3de5ce71d2507" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45113 } } }
I20260926 06:58:06.473134 11569 sys_catalog.cc:458] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [sys.catalog]: This master's current role is: LEADER
I20260926 06:58:06.473073 11568 sys_catalog.cc:455] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e344fc370f574129b6e3de5ce71d2507. Latest consensus state: current_term: 1 leader_uuid: "e344fc370f574129b6e3de5ce71d2507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e344fc370f574129b6e3de5ce71d2507" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45113 } } }
I20260926 06:58:06.473582 11568 sys_catalog.cc:458] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507 [sys.catalog]: This master's current role is: LEADER
I20260926 06:58:06.473728 11576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:58:06.474213 11576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:58:06.475660 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:06.465080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.1.74.126:45113)
0926 06:58:06.465239 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:06.465253 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:06.465287 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:58:06.465669 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:06.465676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:06.465718 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:06.465830 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:06.466615 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.467626 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.468454 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.469067 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.469192 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:06.471641 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:06.471646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:06.471646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:06.471658 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:06.473841 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.474328 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.474332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.474332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.474443 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.474654 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.474655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.474655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.474947 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:06.475078 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:06.475231 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 06:58:06.475421 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260926 06:58:06.475982 11576 catalog_manager.cc:1347] Generated new cluster ID: 4e854c1deb054a96b3887775d1951403
I20260926 06:58:06.476032 11576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:58:06.500186 11576 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:58:06.501013 11576 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:58:06.505892 11576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507: Generated new TSK 0
I20260926 06:58:06.506098 11576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:58:06.538635 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45113
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:58:06.629339 11588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:06.629593 11588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:06.629617 11588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:06.630724 11588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:06.630764 11588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:06.630789 11588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:58:06.632495 11588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:06.632691 11588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:06.632926 11588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:06.638010 11594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:06.638033 11596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:06.638010 11593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:06.638271 11588 server_base.cc:1034] running on GCE node
I20260926 06:58:06.638487 11588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:06.638758 11588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:06.639928 11588 hybrid_clock.cc:648] HybridClock initialized: now 1790405886639914 us; error 48 us; skew 500 ppm
Sep 26 06:58:06 dist-test-slave-sfvt krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:58:06.642314 11588 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:58:06.642740 11588 webserver.cc:460] Webserver started at http://127.1.74.65:44275/ using document root <none> and password file <none>
I20260926 06:58:06.642969 11588 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:06.643039 11588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:06.643190 11588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:06.644263 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "8ebc5114c7bb4df09a3c47dbf248eaae"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "309f1bc88c7f1308f3dde7151bf160f8"
server_key_iv: "109f712137abb0bf7bd78e97aa28bb9d"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.644620 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "8ebc5114c7bb4df09a3c47dbf248eaae"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "309f1bc88c7f1308f3dde7151bf160f8"
server_key_iv: "109f712137abb0bf7bd78e97aa28bb9d"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.645879 11588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:58:06.646574 11603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.646739 11588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:58:06.646831 11588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "8ebc5114c7bb4df09a3c47dbf248eaae"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "309f1bc88c7f1308f3dde7151bf160f8"
server_key_iv: "109f712137abb0bf7bd78e97aa28bb9d"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.646925 11588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:06.653882 11588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:06.655669 11588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:06.655829 11588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:06.656044 11588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:06.656334 11588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:06.656389 11588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.656423 11588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:06.656452 11588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.662060 11588 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:44331
I20260926 06:58:06.662127 11716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:44331 every 8 connection(s)
I20260926 06:58:06.662452 11588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:58:06.663973 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 11588
I20260926 06:58:06.664084 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:58:06.664280 1321 external_mini_cluster.cc:1428] Setting key 1ab531e2a6553922d9f7cd3f31db4ad2
Sep 26 06:58:06 dist-test-slave-sfvt krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
I20260926 06:58:06.672637 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:06.663147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:46407 (local address 127.1.74.126:45113)
0926 06:58:06.663230 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:06.663233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:06.663473 (+ 240us) server_negotiation.cc:407] Connection header received
0926 06:58:06.664041 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:06.664042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:06.664080 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:06.664143 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:06.665427 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.665922 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.667073 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.667405 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.667460 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:06.669760 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:06.669762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:06.669762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:06.669769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:06.671167 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.671640 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.671641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.671641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.671717 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.671964 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.671965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.671965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.672120 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:06.672194 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:06.672459 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 06:58:06.672543 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:58:06.672850 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:06.663106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45113 (local address 127.1.74.65:46407)
0926 06:58:06.663341 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:06.663382 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:06.663912 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:06.664269 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:06.664272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:06.665042 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:06.665318 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:06.665324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.666019 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.666020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:06.666972 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:06.666974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.667483 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.667485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:06.667633 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:06.668126 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:06.668136 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:06.669665 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:06.671327 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:06.671332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:06.671335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:06.671551 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:06.671783 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:06.671784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:06.671785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:06.671896 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:06.672186 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:06.672192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:06.672330 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 06:58:06.672440 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 06:58:06.672587 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":156,"threads_started":1}
I20260926 06:58:06.673357 11717 heartbeater.cc:347] Connected to a master server at 127.1.74.126:45113
I20260926 06:58:06.673467 11717 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:06.673658 11717 heartbeater.cc:511] Master 127.1.74.126:45113 requested a full tablet report, sending...
I20260926 06:58:06.674273 11526 ts_manager.cc:194] Registered new tserver with Master: 8ebc5114c7bb4df09a3c47dbf248eaae (127.1.74.65:44331)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
I20260926 06:58:06.675021 11526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:46407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:58:06.689975 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45113
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:58:06.770248 11722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:06.770447 11722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:06.770470 11722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:06.771654 11722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:06.771694 11722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:06.771744 11722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:58:06.773419 11722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:06.773648 11722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:06.773912 11722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:06.778964 11730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:06.779044 11722 server_base.cc:1034] running on GCE node
W20260926 06:58:06.779062 11727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:06.779014 11728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:06.779482 11722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:06.779760 11722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:06.780946 11722 hybrid_clock.cc:648] HybridClock initialized: now 1790405886780927 us; error 28 us; skew 500 ppm
Sep 26 06:58:06 dist-test-slave-sfvt krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:58:06.783324 11722 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:58:06.783772 11722 webserver.cc:460] Webserver started at http://127.1.74.66:42813/ using document root <none> and password file <none>
I20260926 06:58:06.784037 11722 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:06.784107 11722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:06.784245 11722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:06.785269 11722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "2be6179a1056402583832042533e0697"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "df06b9582d11b2a10f3dc3e4e9bde5c8"
server_key_iv: "24f659f440e47f6716cdd57549e4f69c"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.785660 11722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "2be6179a1056402583832042533e0697"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "df06b9582d11b2a10f3dc3e4e9bde5c8"
server_key_iv: "24f659f440e47f6716cdd57549e4f69c"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.786907 11722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:58:06.787647 11737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.787798 11722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.787894 11722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "2be6179a1056402583832042533e0697"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "df06b9582d11b2a10f3dc3e4e9bde5c8"
server_key_iv: "24f659f440e47f6716cdd57549e4f69c"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.787969 11722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:06.800652 11722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:06.802692 11722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:06.802860 11722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:06.803103 11722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:06.803524 11722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:06.803581 11722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.803624 11722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:06.803651 11722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.809731 11722 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:39761
I20260926 06:58:06.809780 11850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:39761 every 8 connection(s)
I20260926 06:58:06.810110 11722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:58:06.814678 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 11722
I20260926 06:58:06.814798 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:58:06.815047 1321 external_mini_cluster.cc:1428] Setting key f52c9372073b988b2517e9cec397cfe2
Sep 26 06:58:06 dist-test-slave-sfvt krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:58:06.820451 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:06.810881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:44661 (local address 127.1.74.126:45113)
0926 06:58:06.810974 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:06.810976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:06.811107 (+ 131us) server_negotiation.cc:407] Connection header received
0926 06:58:06.811695 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:06.811696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:06.811706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:06.811754 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:06.812769 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.813262 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.814317 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.814487 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.814539 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:06.817509 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:06.817510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:06.817511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:06.817520 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:06.818936 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.819431 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.819432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.819432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.819489 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.819771 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.819776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.819776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.819879 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:06.819952 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:06.820252 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 06:58:06.820333 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 06:58:06.820641 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:06.810797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45113 (local address 127.1.74.66:44661)
0926 06:58:06.810959 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:06.811012 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:06.811595 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:06.811827 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:06.811830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:06.812451 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:06.812690 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:06.812695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.813362 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.813363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:06.814241 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:06.814243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.814594 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.814596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:06.814763 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:06.815480 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:06.815491 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:06.817400 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:06.819049 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:06.819056 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:06.819058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:06.819349 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:06.819556 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:06.819557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:06.819557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:06.819695 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:06.819955 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:06.819962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:06.820103 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 06:58:06.820217 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 06:58:06.820379 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260926 06:58:06.821288 11851 heartbeater.cc:347] Connected to a master server at 127.1.74.126:45113
I20260926 06:58:06.821372 11851 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:06.821568 11851 heartbeater.cc:511] Master 127.1.74.126:45113 requested a full tablet report, sending...
I20260926 06:58:06.821985 11526 ts_manager.cc:194] Registered new tserver with Master: 2be6179a1056402583832042533e0697 (127.1.74.66:39761)
I20260926 06:58:06.822368 11526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:44661
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:58:06.843817 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45113
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:58:06.926276 11856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:06.926532 11856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:06.926570 11856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:06.927719 11856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:06.927793 11856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:06.927837 11856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:58:06.929239 11856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:06.929507 11856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:06.929736 11856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:06.934926 11864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:06.935082 11856 server_base.cc:1034] running on GCE node
W20260926 06:58:06.935041 11861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:06.934988 11862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:06.935478 11856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:06.935683 11856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:06.936823 11856 hybrid_clock.cc:648] HybridClock initialized: now 1790405886936811 us; error 29 us; skew 500 ppm
Sep 26 06:58:06 dist-test-slave-sfvt krb5kdc[11481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:58:06.944691 11856 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:58:06.945191 11856 webserver.cc:460] Webserver started at http://127.1.74.67:33005/ using document root <none> and password file <none>
I20260926 06:58:06.945428 11856 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:06.945497 11856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:06.945650 11856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:58:06.946699 11856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "0ee10afecf164fc78d67656389702591"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "78a60136b7be6f927d6cde3f8f2f8a82"
server_key_iv: "f8dcbf3fef449e41dbc3f7299141e480"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.947072 11856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "0ee10afecf164fc78d67656389702591"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "78a60136b7be6f927d6cde3f8f2f8a82"
server_key_iv: "f8dcbf3fef449e41dbc3f7299141e480"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.948431 11856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:58:06.949177 11871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.949333 11856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.949419 11856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "0ee10afecf164fc78d67656389702591"
format_stamp: "Formatted at 2026-09-26 06:58:06 on dist-test-slave-sfvt"
server_key: "78a60136b7be6f927d6cde3f8f2f8a82"
server_key_iv: "f8dcbf3fef449e41dbc3f7299141e480"
server_key_version: "encryptionkey@0"
I20260926 06:58:06.949472 11856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:06.963482 11856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:06.965790 11856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:06.965903 11856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:06.966101 11856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:06.966346 11856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:06.966372 11856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.966393 11856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:06.966406 11856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:06.972178 11856 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:43195
I20260926 06:58:06.972255 11984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:43195 every 8 connection(s)
I20260926 06:58:06.972620 11856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:58:06.978377 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 11856
I20260926 06:58:06.978487 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:58:06.978652 1321 external_mini_cluster.cc:1428] Setting key 528c2b1c9d9445b85746f415a505a0a8
Sep 26 06:58:06 dist-test-slave-sfvt krb5kdc[11481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405886, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:58:06.982218 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:06.973338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:33507 (local address 127.1.74.126:45113)
0926 06:58:06.973408 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:06.973409 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:06.973638 (+ 229us) server_negotiation.cc:407] Connection header received
0926 06:58:06.974234 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:06.974235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:06.974248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:06.974302 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:06.975377 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.975903 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.976901 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.977086 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.977136 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:06.979662 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:06.979663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:06.979663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:06.979669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:06.981054 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.981438 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.981439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.981439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.981486 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.981677 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.981677 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.981678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.981771 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:06.981820 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:06.982099 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 06:58:06.982156 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:58:06.982465 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:06.973316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45113 (local address 127.1.74.67:33507)
0926 06:58:06.973489 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:06.973532 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:06.974140 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:06.974369 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:06.974372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:06.975045 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:06.975237 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:06.975241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.976008 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.976009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:06.976813 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:06.976816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.977181 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.977182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:06.977297 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:06.977764 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:06.977773 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:06.979591 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:06.981133 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:06.981140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:06.981143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:06.981366 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:06.981528 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:06.981528 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:06.981528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:06.981625 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:06.981847 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:06.981852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:06.981971 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 06:58:06.982076 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 06:58:06.982194 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260926 06:58:06.982899 11985 heartbeater.cc:347] Connected to a master server at 127.1.74.126:45113
I20260926 06:58:06.983006 11985 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:06.983189 11985 heartbeater.cc:511] Master 127.1.74.126:45113 requested a full tablet report, sending...
I20260926 06:58:06.983563 11526 ts_manager.cc:194] Registered new tserver with Master: 0ee10afecf164fc78d67656389702591 (127.1.74.67:43195)
I20260926 06:58:06.983943 11526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:33507
I20260926 06:58:06.990435 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:58:06.997501 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:06.991687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.1.74.126:45113)
0926 06:58:06.991768 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:06.991770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:06.991860 (+ 90us) server_negotiation.cc:407] Connection header received
0926 06:58:06.991947 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:06.991948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:06.991957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:06.992033 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:06.992587 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.993119 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.993833 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:06.994039 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:06.994122 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:06.995210 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:06.995212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:06.995212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:06.995219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:06.996368 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.996724 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.996725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.996725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.996776 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:06.996972 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:06.996973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:06.996973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:06.997116 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:06.997236 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:06.997329 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 06:58:06.997403 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 06:58:06.998857 11526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:58:07.000453 11526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:58:07.006433 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.002473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:39761 (local address 127.0.0.1:48746)
0926 06:58:07.002735 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:07.002775 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:07.002835 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:07.003094 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:07.003097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:07.003099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:07.003164 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.003166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.004208 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.004209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.005234 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.005236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.006110 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.006110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.006187 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.006200 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 06:58:07.006298 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 06:58:07.006349 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":60,"threads_started":1}
I20260926 06:58:07.006445 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.002605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.1.74.66:39761)
0926 06:58:07.002924 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:07.002930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:07.002940 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:58:07.002976 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:07.002978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:07.002993 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:07.003087 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:07.003234 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.004097 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.005328 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.006043 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.006110 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.006172 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:07.006314 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 06:58:07.006353 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":84,"threads_started":1}
I20260926 06:58:07.006898 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.002766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44331 (local address 127.0.0.1:52028)
0926 06:58:07.003121 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:07.003131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:07.003183 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:07.003473 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:07.003474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:07.003476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:07.003531 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.003533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.004814 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.004815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.005761 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.005764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.006639 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.006641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.006705 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.006710 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:58:07.006767 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 06:58:07.006812 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":51,"thread_start_us":41,"threads_started":1}
I20260926 06:58:07.006923 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.002948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52028 (local address 127.1.74.65:44331)
0926 06:58:07.003109 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:07.003114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:07.003173 (+ 59us) server_negotiation.cc:407] Connection header received
0926 06:58:07.003299 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:07.003302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:07.003318 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:07.003388 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:07.003596 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.004695 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.005849 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.006543 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.006616 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.006665 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:07.006796 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 06:58:07.006837 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260926 06:58:07.007299 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.002632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:43195 (local address 127.0.0.1:47300)
0926 06:58:07.002772 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:07.002785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:07.002840 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:07.003463 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:07.003464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:07.003466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:07.003518 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.003519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.004809 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.004809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.005974 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.005976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.007051 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.007052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.007122 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.007126 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:58:07.007176 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:58:07.007211 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":45,"thread_start_us":56,"threads_started":1}
I20260926 06:58:07.007596 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.002732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.1.74.67:43195)
0926 06:58:07.003221 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:07.003225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:07.003235 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:58:07.003290 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:07.003292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:07.003307 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:07.003378 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:07.003595 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.004695 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.006066 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.006881 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.006958 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.007014 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:07.007469 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 06:58:07.007511 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":77,"threads_started":1}
I20260926 06:58:07.007556 11785 tablet_service.cc:1467] Processing CreateTablet for tablet 577e2b01c4fc459faad58cf35af57a1b (DEFAULT_TABLE table=test-table [id=bb39dea7d25d40c98146c15992967bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:58:07.007925 11785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577e2b01c4fc459faad58cf35af57a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:07.007822 11651 tablet_service.cc:1467] Processing CreateTablet for tablet 577e2b01c4fc459faad58cf35af57a1b (DEFAULT_TABLE table=test-table [id=bb39dea7d25d40c98146c15992967bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:58:07.008127 11651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577e2b01c4fc459faad58cf35af57a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:07.008175 11919 tablet_service.cc:1467] Processing CreateTablet for tablet 577e2b01c4fc459faad58cf35af57a1b (DEFAULT_TABLE table=test-table [id=bb39dea7d25d40c98146c15992967bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:58:07.008435 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 577e2b01c4fc459faad58cf35af57a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:07.010780 11999 tablet_bootstrap.cc:492] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: Bootstrap starting.
I20260926 06:58:07.011320 12000 tablet_bootstrap.cc:492] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae: Bootstrap starting.
I20260926 06:58:07.011511 11999 tablet_bootstrap.cc:654] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:07.011550 12001 tablet_bootstrap.cc:492] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: Bootstrap starting.
I20260926 06:58:07.012060 12001 tablet_bootstrap.cc:654] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:07.012066 12000 tablet_bootstrap.cc:654] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:07.012071 11999 log.cc:824] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:07.012688 12001 log.cc:824] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:07.012693 12000 log.cc:824] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:07.012882 11999 tablet_bootstrap.cc:492] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: No bootstrap required, opened a new log
I20260926 06:58:07.013005 11999 ts_tablet_manager.cc:1397] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:58:07.013370 12000 tablet_bootstrap.cc:492] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae: No bootstrap required, opened a new log
I20260926 06:58:07.013451 12001 tablet_bootstrap.cc:492] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: No bootstrap required, opened a new log
I20260926 06:58:07.013455 12000 ts_tablet_manager.cc:1397] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:58:07.013561 12001 ts_tablet_manager.cc:1397] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:58:07.014752 12000 raft_consensus.cc:348] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } }
I20260926 06:58:07.014919 12000 raft_consensus.cc:374] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:07.014966 12000 raft_consensus.cc:729] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebc5114c7bb4df09a3c47dbf248eaae, State: Initialized, Role: FOLLOWER
I20260926 06:58:07.014986 11999 raft_consensus.cc:348] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } }
I20260926 06:58:07.015097 12000 consensus_queue.cc:260] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } }
I20260926 06:58:07.015138 11999 raft_consensus.cc:374] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:07.015175 11999 raft_consensus.cc:729] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be6179a1056402583832042533e0697, State: Initialized, Role: FOLLOWER
I20260926 06:58:07.015352 11999 consensus_queue.cc:260] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } }
I20260926 06:58:07.015410 11717 heartbeater.cc:503] Master 127.1.74.126:45113 was elected leader, sending a full tablet report...
I20260926 06:58:07.015537 12000 ts_tablet_manager.cc:1428] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:58:07.015486 12001 raft_consensus.cc:348] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } }
I20260926 06:58:07.015771 11999 ts_tablet_manager.cc:1428] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 06:58:07.015829 11851 heartbeater.cc:503] Master 127.1.74.126:45113 was elected leader, sending a full tablet report...
I20260926 06:58:07.015792 12001 raft_consensus.cc:374] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:07.015879 12001 raft_consensus.cc:729] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee10afecf164fc78d67656389702591, State: Initialized, Role: FOLLOWER
I20260926 06:58:07.016032 12001 consensus_queue.cc:260] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } }
I20260926 06:58:07.016341 12001 ts_tablet_manager.cc:1428] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:58:07.016462 11985 heartbeater.cc:503] Master 127.1.74.126:45113 was elected leader, sending a full tablet report...
W20260926 06:58:07.060640 11852 tablet.cc:2369] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:58:07.143653 12007 raft_consensus.cc:482] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:58:07.143806 12007 raft_consensus.cc:504] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } }
I20260926 06:58:07.144250 12007 leader_election.cc:288] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2be6179a1056402583832042533e0697 (127.1.74.66:39761), 8ebc5114c7bb4df09a3c47dbf248eaae (127.1.74.65:44331)
I20260926 06:58:07.147238 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.144570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44331 (local address 127.1.74.67:46581)
0926 06:58:07.144734 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:07.144744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:07.144799 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:07.144928 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:07.144929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:07.144930 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:07.144992 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.144993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.145570 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.145572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.146372 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.146374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.146982 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.146983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.147053 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.147059 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:58:07.147107 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:58:07.147137 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260926 06:58:07.147274 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.144642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:46581 (local address 127.1.74.65:44331)
0926 06:58:07.144706 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:07.144708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:07.144803 (+ 95us) server_negotiation.cc:407] Connection header received
0926 06:58:07.144849 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:07.144850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:07.144862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:07.144918 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:07.145053 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.145499 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.146497 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.146889 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.146979 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.146992 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:07.147135 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 06:58:07.147182 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:58:07.147367 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.144532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:52055 (local address 127.1.74.66:39761)
0926 06:58:07.144602 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:07.144604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:07.144615 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:58:07.144645 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:07.144646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:07.144657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:07.144715 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:07.144932 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.145501 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.146471 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.146888 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.146999 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.147009 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:07.147216 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 06:58:07.147288 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:58:07.147342 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.144410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:39761 (local address 127.1.74.67:52055)
0926 06:58:07.144486 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:07.144498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:07.144569 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:07.144751 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:07.144752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:07.144753 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:07.144849 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.144851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.145616 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.145617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.146374 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:07.146376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.147029 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.147029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:07.147135 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.147140 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:58:07.147195 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:58:07.147231 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":57}
I20260926 06:58:07.147825 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577e2b01c4fc459faad58cf35af57a1b" candidate_uuid: "0ee10afecf164fc78d67656389702591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be6179a1056402583832042533e0697" is_pre_election: true
I20260926 06:58:07.147830 11671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577e2b01c4fc459faad58cf35af57a1b" candidate_uuid: "0ee10afecf164fc78d67656389702591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" is_pre_election: true
I20260926 06:58:07.147986 11805 raft_consensus.cc:2454] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee10afecf164fc78d67656389702591 in term 0.
I20260926 06:58:07.147987 11671 raft_consensus.cc:2454] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee10afecf164fc78d67656389702591 in term 0.
I20260926 06:58:07.148196 11873 leader_election.cc:302] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee10afecf164fc78d67656389702591, 2be6179a1056402583832042533e0697; no voters:
I20260926 06:58:07.148360 12007 raft_consensus.cc:2781] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:58:07.148437 12007 raft_consensus.cc:482] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:58:07.148464 12007 raft_consensus.cc:3037] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:58:07.149348 12007 raft_consensus.cc:504] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } }
I20260926 06:58:07.149444 12007 leader_election.cc:288] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [CANDIDATE]: Term 1 election: Requested vote from peers 2be6179a1056402583832042533e0697 (127.1.74.66:39761), 8ebc5114c7bb4df09a3c47dbf248eaae (127.1.74.65:44331)
I20260926 06:58:07.149588 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577e2b01c4fc459faad58cf35af57a1b" candidate_uuid: "0ee10afecf164fc78d67656389702591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be6179a1056402583832042533e0697"
I20260926 06:58:07.149617 11671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "577e2b01c4fc459faad58cf35af57a1b" candidate_uuid: "0ee10afecf164fc78d67656389702591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae"
I20260926 06:58:07.149659 11805 raft_consensus.cc:3037] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:58:07.149695 11671 raft_consensus.cc:3037] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:58:07.150463 11805 raft_consensus.cc:2454] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee10afecf164fc78d67656389702591 in term 1.
I20260926 06:58:07.150568 11671 raft_consensus.cc:2454] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee10afecf164fc78d67656389702591 in term 1.
I20260926 06:58:07.150628 11873 leader_election.cc:302] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee10afecf164fc78d67656389702591, 2be6179a1056402583832042533e0697; no voters:
I20260926 06:58:07.150704 12007 raft_consensus.cc:2781] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:58:07.150852 12007 raft_consensus.cc:686] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 1 LEADER]: Becoming Leader. State: Replica: 0ee10afecf164fc78d67656389702591, State: Running, Role: LEADER
I20260926 06:58:07.150967 12007 consensus_queue.cc:237] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } }
I20260926 06:58:07.151913 11525 catalog_manager.cc:5495] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee10afecf164fc78d67656389702591 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "0ee10afecf164fc78d67656389702591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ee10afecf164fc78d67656389702591" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 43195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 } health_report { overall_health: UNKNOWN } } }
W20260926 06:58:07.163141 11718 tablet.cc:2369] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:58:07.189203 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.186956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.1.74.67:43195)
0926 06:58:07.187020 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:07.187022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:07.187029 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:58:07.187064 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:07.187065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:07.187082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:07.187143 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:58:07.187318 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.187790 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.188498 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:07.188678 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:07.188726 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:07.188903 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:58:07.189007 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:58:07.189067 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:07.189111 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 06:58:07.189129 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:58:07.191056 11671 raft_consensus.cc:1261] T 577e2b01c4fc459faad58cf35af57a1b P 8ebc5114c7bb4df09a3c47dbf248eaae [term 1 FOLLOWER]: Refusing update from remote peer 0ee10afecf164fc78d67656389702591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:58:07.191056 11805 raft_consensus.cc:1261] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [term 1 FOLLOWER]: Refusing update from remote peer 0ee10afecf164fc78d67656389702591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:58:07.191425 12007 consensus_queue.cc:1035] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be6179a1056402583832042533e0697" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:58:07.191573 12009 consensus_queue.cc:1035] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebc5114c7bb4df09a3c47dbf248eaae" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:58:07.194522 12015 mvcc.cc:205] Tried to move back new op lower bound from 7333502513931571200 to 7333502513770950656. Current Snapshot: MvccSnapshot[applied={T|T < 7333502513931571200}]
I20260926 06:58:07.196390 11526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:58:07.196498 11526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:58:07.197253 11526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e344fc370f574129b6e3de5ce71d2507: Sending DeleteTablet for 3 replicas of tablet 577e2b01c4fc459faad58cf35af57a1b
I20260926 06:58:07.197614 11785 tablet_service.cc:1514] Processing DeleteTablet for tablet 577e2b01c4fc459faad58cf35af57a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:58:07 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:48746
I20260926 06:58:07.197631 11651 tablet_service.cc:1514] Processing DeleteTablet for tablet 577e2b01c4fc459faad58cf35af57a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:58:07 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:52028
I20260926 06:58:07.197861 12023 tablet_replica.cc:330] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: stopping tablet replica
I20260926 06:58:07.197839 11919 tablet_service.cc:1514] Processing DeleteTablet for tablet 577e2b01c4fc459faad58cf35af57a1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:58:07 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:47300
I20260926 06:58:07.197958 12023 raft_consensus.cc:2229] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:58:07.198058 12023 raft_consensus.cc:2258] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:58:07.198153 12024 tablet_replica.cc:330] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: stopping tablet replica
I20260926 06:58:07.198235 12024 raft_consensus.cc:2229] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:58:07.198315 12024 raft_consensus.cc:2258] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:58:07.198414 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 11588
I20260926 06:58:07.198734 12024 ts_tablet_manager.cc:1905] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:58:07.200518 12024 ts_tablet_manager.cc:1918] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:58:07.200618 12024 log.cc:1196] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/577e2b01c4fc459faad58cf35af57a1b
I20260926 06:58:07.200866 12024 ts_tablet_manager.cc:1939] T 577e2b01c4fc459faad58cf35af57a1b P 0ee10afecf164fc78d67656389702591: Deleting consensus metadata
I20260926 06:58:07.201246 11511 catalog_manager.cc:4841] TS 0ee10afecf164fc78d67656389702591 (127.1.74.67:43195): tablet 577e2b01c4fc459faad58cf35af57a1b (table test-table [id=bb39dea7d25d40c98146c15992967bbc]) successfully deleted
I20260926 06:58:07.203595 12023 ts_tablet_manager.cc:1905] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:58:07.205075 12023 ts_tablet_manager.cc:1918] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:58:07.205163 12023 log.cc:1196] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/577e2b01c4fc459faad58cf35af57a1b
I20260926 06:58:07.205463 12023 ts_tablet_manager.cc:1939] T 577e2b01c4fc459faad58cf35af57a1b P 2be6179a1056402583832042533e0697: Deleting consensus metadata
I20260926 06:58:07.205899 11511 catalog_manager.cc:4841] TS 2be6179a1056402583832042533e0697 (127.1.74.66:39761): tablet 577e2b01c4fc459faad58cf35af57a1b (table test-table [id=bb39dea7d25d40c98146c15992967bbc]) successfully deleted
W20260926 06:58:07.206498 11511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:44331 (error 108)
I20260926 06:58:07.206691 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 11722
I20260926 06:58:07.207489 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:07.207361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44331 (local address 127.0.0.1:52042)
0926 06:58:07.207411 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:07.207446 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:58:07.207612 11511 catalog_manager.cc:4584] TS 8ebc5114c7bb4df09a3c47dbf248eaae (127.1.74.65:44331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 577e2b01c4fc459faad58cf35af57a1b: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44331: connect: Connection refused (error 111)
I20260926 06:58:07.212782 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 11856
I20260926 06:58:07.217674 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 11494
2026-09-26T06:58:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2021 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:58:07 dist-test-slave-sfvt krb5kdc[12028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:58:07 dist-test-slave-sfvt krb5kdc[12028](info): set up 2 sockets
Sep 26 06:58:07 dist-test-slave-sfvt krb5kdc[12028](info): commencing operation
krb5kdc: starting...
W20260926 06:58:08.256309 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:58:08 dist-test-slave-sfvt krb5kdc[12028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:58:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.74.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.74.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/rangeradmin_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/rangeradmin_127.1.74.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.74.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.74.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/rangerlookup_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/rangerlookup_127.1.74.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.74.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.74.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/HTTP_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/HTTP_127.1.74.84@KRBTEST.COM.keytab.
I20260926 06:58:08.420342 1321 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taske4hwn4/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/postgres -l logfile start
2026-09-26 06:58:11.043 UTC [12058] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 06:58:11.043 UTC [12058] LOG: listening on IPv4 address "127.1.74.84", port 50079
2026-09-26 06:58:11.047 UTC [12058] LOG: listening on Unix socket "/tmp/.s.PGSQL.50079"
2026-09-26 06:58:11.069 UTC [12061] LOG: database system was shut down at 2026-09-26 06:58:09 UTC
2026-09-26 06:58:11.073 UTC [12058] LOG: database system is ready to accept connections
I20260926 06:58:12.016127 1321 mini_postgres.cc:96] Postgres bound to 50079
2026-09-26 06:58:12.022 UTC [12069] FATAL: database "slave" does not exist
127.1.74.84:50079 - accepting connections
I20260926 06:58:12.022589 1321 mini_ranger.cc:161] Starting Ranger...
I20260926 06:58:12.037147 1321 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 06:58:12.545192 1321 mini_ranger.cc:87] Created ranger Postgres database
I20260926 06:58:12.545254 1321 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-admin
2026-09-26 06:58:12,803 [I] DB FLAVOR :POSTGRES
2026-09-26 06:58:12,804 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:58:12,804 [I] Checking connection..
2026-09-26 06:58:12,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 06:58:13,052 [I] Checking connection passed.
2026-09-26 06:58:13,052 [I] --------- Verifying version history table ---------
2026-09-26 06:58:13,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:58:13,312 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:58:13,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:58:13,570 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:58:13,570 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 06:58:13,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 06:58:13.799 UTC [12133] WARNING: there is no transaction in progress
2026-09-26 06:58:13,805 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 06:58:13,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:58:14,039 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 06:58:14,039 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:58:14,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 06:58:14,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 06:58:14,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-sfvt.c.gcp-upstream.internal','N') ;"
2026-09-26 06:58:14,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:58:15,028 [I] Table x_portal_user does not exist in database ranger
2026-09-26 06:58:15,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:58:15,256 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 06:58:15,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:58:15,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:58:15,713 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 06:58:15,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 06:58:16.691 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.707 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.723 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.739 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.755 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.773 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.828 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.835 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.850 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.861 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.871 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.886 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.897 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.904 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.912 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:16.919 UTC [12241] WARNING: there is no transaction in progress
2026-09-26 06:58:17,561 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 06:58:17,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 06:58:17,796 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 06:58:17,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:58:18,022 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 06:58:18,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:58:18,263 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 06:58:18,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:58:18,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:58:18,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-sfvt.c.gcp-upstream.internal';"
2026-09-26 06:58:18,988 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 06:58:18,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 06:58:19,227 [I] DB_PATCHES have already been applied
I20260926 06:58:19.233147 1321 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-admin:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taske4hwn4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 06:58:19.233229 1321 mini_ranger.cc:193] Using host: 127.1.74.84
I20260926 06:58:19.236183 1321 mini_ranger.cc:227] Ranger admin URL: http://127.1.74.84:34545
Sep 26, 2026 6:58:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:58:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:58:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 6:58:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 6:58:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 6:58:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 6:58:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 06:58:20.341085 1321 mini_ranger.cc:160] Time spent starting Ranger: real 8.318s user 0.000s sys 0.008s
Sep 26, 2026 6:58:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 6:58:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 6:58:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 6:58:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 6:58:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 06:58:40.824226 1321 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.74.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.74.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/rangerkms_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/rangerkms_127.1.74.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.74.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.74.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.74.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/HTTP_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.74.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/HTTP_127.1.74.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 06:58:40 dist-test-slave-sfvt krb5kdc[12028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 06:58:40.914773 1321 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 06:58:40.924957 1321 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 06:58:41.854387 1321 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 06:58:41.854453 1321 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-kms
I20260926 06:58:41.854466 1321 mini_ranger_kms.cc:227] Using postgres at 127.1.74.84:50079
2026-09-26 06:58:42,031 [I] DB FLAVOR :POSTGRES
2026-09-26 06:58:42,031 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:58:42,031 [I] Checking connection
2026-09-26 06:58:42,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 06:58:42,288 [I] connection success
2026-09-26 06:58:42,288 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 06:58:42,288 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 06:58:42,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 06:58:42,517 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 06:58:42,517 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:58:42,517 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 06:58:42,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:50079/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 06:58:42,787 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 06:58:43.221020 1321 mini_ranger_kms.cc:310] Created kms service
I20260926 06:58:43.394248 1321 mini_ranger_kms.cc:326] Created kudu user
I20260926 06:58:43.434432 1321 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 06:58:43 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405920, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:58:43.780411 1321 mini_ranger_kms.cc:384] Added ranger policy
I20260926 06:58:43.780521 1321 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-kms:/tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 06:58:43.780542 1321 mini_ranger_kms.cc:240] Using host: 127.1.74.84
I20260926 06:58:43.784011 1321 mini_ranger_kms.cc:276] Ranger KMS PID: 12421
I20260926 06:58:43.784116 1321 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.74.84:52789
Sep 26, 2026 6:58:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:58:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:58:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 6:58:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 6:58:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 6:58:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 06:58:46.025504 1321 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.111s user 0.008s sys 0.150s
I20260926 06:58:46.025601 1321 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 06:58:46.025673 1321 mini_ranger_kms.cc:401] 127.1.74.84:52789/kms/v1/keys
Sep 26 06:58:46 dist-test-slave-sfvt krb5kdc[12028](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790405926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.74.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 06:58:46 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.74.84@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
Sep 26 06:58:46 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.74.84@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
Sep 26 06:58:49 dist-test-slave-sfvt krb5kdc[12028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:58:49.234602 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46521
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:46521
--ranger_config_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:52789/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:58:49.316123 12472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:49.316375 12472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:49.316413 12472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:49.317586 12472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:58:49.317654 12472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:49.317677 12472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:49.317698 12472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:58:49.317727 12472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:58:49.319379 12472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.74.84:52789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:46521
--ranger_config_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:49.319649 12472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:49.319885 12472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:49.326224 12477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:49.326370 12478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:49.326362 12472 server_base.cc:1034] running on GCE node
W20260926 06:58:49.326256 12480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:49.326742 12472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:49.327041 12472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:49.328200 12472 hybrid_clock.cc:648] HybridClock initialized: now 1790405929328172 us; error 44 us; skew 500 ppm
Sep 26 06:58:49 dist-test-slave-sfvt krb5kdc[12028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405929, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:58:49.333567 12472 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:58:49.334086 12472 webserver.cc:460] Webserver started at http://127.1.74.126:38905/ using document root <none> and password file <none>
I20260926 06:58:49.334309 12472 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:49.334367 12472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:49.334501 12472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:58:49 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405929, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:58:49.387162 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "76d033b8a7b543f9aaae05abb3249a01"
format_stamp: "Formatted at 2026-09-26 06:58:49 on dist-test-slave-sfvt"
server_key: "02a2ccd1c66380a9a367c056e222cf08"
server_key_iv: "2192721a97ec7a0d8c67b56cbbed994d"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:49.387691 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "76d033b8a7b543f9aaae05abb3249a01"
format_stamp: "Formatted at 2026-09-26 06:58:49 on dist-test-slave-sfvt"
server_key: "02a2ccd1c66380a9a367c056e222cf08"
server_key_iv: "2192721a97ec7a0d8c67b56cbbed994d"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:49.389556 12472 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 06:58:49.403079 12487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:49.403314 12472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:58:49.403411 12472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "76d033b8a7b543f9aaae05abb3249a01"
format_stamp: "Formatted at 2026-09-26 06:58:49 on dist-test-slave-sfvt"
server_key: "02a2ccd1c66380a9a367c056e222cf08"
server_key_iv: "2192721a97ec7a0d8c67b56cbbed994d"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:49.403501 12472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:49.444485 12472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:49.446564 12472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:49.446731 12472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:49.452052 12472 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:46521
I20260926 06:58:49.452131 12539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:46521 every 8 connection(s)
I20260926 06:58:49.452625 12472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:58:49.453337 12540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:49.455608 12540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01: Bootstrap starting.
I20260926 06:58:49.456161 12540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:49.456437 12540 log.cc:824] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:49.457167 12540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01: No bootstrap required, opened a new log
I20260926 06:58:49.459141 12540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d033b8a7b543f9aaae05abb3249a01" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46521 } }
I20260926 06:58:49.459326 12540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:49.459375 12540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d033b8a7b543f9aaae05abb3249a01, State: Initialized, Role: FOLLOWER
I20260926 06:58:49.459532 12540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d033b8a7b543f9aaae05abb3249a01" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46521 } }
I20260926 06:58:49.459612 12540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:58:49.459651 12540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:58:49.459700 12540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:58:49.460219 12540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d033b8a7b543f9aaae05abb3249a01" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46521 } }
I20260926 06:58:49.460333 12540 leader_election.cc:302] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d033b8a7b543f9aaae05abb3249a01; no voters:
I20260926 06:58:49.460515 12540 leader_election.cc:288] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:58:49.460579 12543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:58:49.460949 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 12472
I20260926 06:58:49.461040 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:58:49.461192 1321 external_mini_cluster.cc:1428] Setting key 2888e6fbec49aa83894dea7cc808e522
I20260926 06:58:49.460757 12540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:58:49.460779 12543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [term 1 LEADER]: Becoming Leader. State: Replica: 76d033b8a7b543f9aaae05abb3249a01, State: Running, Role: LEADER
I20260926 06:58:49.461544 12543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d033b8a7b543f9aaae05abb3249a01" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46521 } }
I20260926 06:58:49.462550 12545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d033b8a7b543f9aaae05abb3249a01. Latest consensus state: current_term: 1 leader_uuid: "76d033b8a7b543f9aaae05abb3249a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d033b8a7b543f9aaae05abb3249a01" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46521 } } }
I20260926 06:58:49.462687 12545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [sys.catalog]: This master's current role is: LEADER
I20260926 06:58:49.462891 12544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d033b8a7b543f9aaae05abb3249a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d033b8a7b543f9aaae05abb3249a01" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46521 } } }
I20260926 06:58:49.462914 12560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:58:49.462925 12540 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 06:58:49.463068 12544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01 [sys.catalog]: This master's current role is: LEADER
I20260926 06:58:49.463567 12560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:58:49.469566 12560 catalog_manager.cc:1347] Generated new cluster ID: 88df3a41b0464862b73484eb0312a33b
I20260926 06:58:49.469663 12560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:58:49 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:58:49.477754 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:49.461898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.1.74.126:46521)
0926 06:58:49.463155 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:49.463165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:49.463388 (+ 223us) server_negotiation.cc:407] Connection header received
0926 06:58:49.463611 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:49.463620 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:49.463690 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:49.464626 (+ 936us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:49.464757 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:49.469224 (+ 4467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:49.470425 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:49.471009 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:49.473731 (+ 2722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:49.473838 (+ 107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:49.473841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:49.473841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:49.473850 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:49.475689 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:49.476253 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:49.476256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:49.476256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:49.476374 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:49.476580 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:49.476581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:49.476581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:49.476852 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:49.477363 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:49.477439 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 06:58:49.477545 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":61,"threads_started":1}
I20260926 06:58:49.490046 12560 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:58:49.490792 12560 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:58:49.499683 12560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01: Generated new TSK 0
I20260926 06:58:49.499931 12560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:58:50.162369 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:52789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46521
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:58:50.251782 12585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:50.252022 12585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:50.252063 12585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:50.253914 12585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:50.253981 12585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:50.254024 12585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:58:50.256232 12585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.74.84:52789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:50.256512 12585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:50.256806 12585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:50.263336 12590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:50.264142 12593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:50.265780 12591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:50.268375 12585 server_base.cc:1034] running on GCE node
I20260926 06:58:50.268797 12585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:50.269207 12585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:50.270390 12585 hybrid_clock.cc:648] HybridClock initialized: now 1790405930270381 us; error 49 us; skew 500 ppm
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:58:50.273583 12585 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:58:50.274097 12585 webserver.cc:460] Webserver started at http://127.1.74.65:34325/ using document root <none> and password file <none>
I20260926 06:58:50.274360 12585 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:50.274411 12585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:50.274554 12585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:58:50.290158 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "2f1be6219bbc48429ac64d9181d3fefb"
format_stamp: "Formatted at 2026-09-26 06:58:50 on dist-test-slave-sfvt"
server_key: "56cf9ef7ef28b2d53a9c8d51cc7f6586"
server_key_iv: "3fe5c48e17326b50dd854f6e537948bb"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:50.290613 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "2f1be6219bbc48429ac64d9181d3fefb"
format_stamp: "Formatted at 2026-09-26 06:58:50 on dist-test-slave-sfvt"
server_key: "56cf9ef7ef28b2d53a9c8d51cc7f6586"
server_key_iv: "3fe5c48e17326b50dd854f6e537948bb"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:50.294001 12585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 06:58:50.311900 12600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:50.312255 12585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:58:50.312378 12585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "2f1be6219bbc48429ac64d9181d3fefb"
format_stamp: "Formatted at 2026-09-26 06:58:50 on dist-test-slave-sfvt"
server_key: "56cf9ef7ef28b2d53a9c8d51cc7f6586"
server_key_iv: "3fe5c48e17326b50dd854f6e537948bb"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:50.312484 12585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:50.328307 12585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:50.331012 12585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:50.331173 12585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:50.331480 12585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:50.331830 12585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:50.331873 12585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:50.331903 12585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:50.331928 12585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:50.339764 12585 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:37155
I20260926 06:58:50.339890 12713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:37155 every 8 connection(s)
I20260926 06:58:50.340175 12585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:58:50.348481 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 12585
I20260926 06:58:50.348587 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:58:50.348764 1321 external_mini_cluster.cc:1428] Setting key 7ce5b4ddc50298ff10b6a77be6554fac
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:58:50.354604 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.340927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:46521 (local address 127.1.74.65:35051)
0926 06:58:50.341125 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:50.341172 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:50.341789 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:50.343925 (+ 2136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:50.343930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:50.344171 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:50.344444 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.344451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.345508 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.345509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.346784 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.346786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.347341 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.347343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.347497 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.348050 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:50.348060 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:50.349761 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:50.352760 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:50.352766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:50.352769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:50.353097 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:50.353348 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:50.353349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:50.353349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:50.353496 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:50.353805 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:50.353812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:50.353999 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 06:58:50.354120 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 06:58:50.354305 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260926 06:58:50.355103 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.341210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:35051 (local address 127.1.74.126:46521)
0926 06:58:50.343690 (+ 2480us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:50.343693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:50.343702 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:58:50.343731 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:50.343732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:50.343744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:50.344553 (+ 809us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:50.344594 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.345389 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.346887 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.347195 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.347312 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.349870 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:50.349872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:50.349872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:50.349880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:50.352666 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:50.353184 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:50.353186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:50.353187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:50.353257 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:50.353572 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:50.353572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:50.353573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:50.353719 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:50.353778 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:50.354867 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0926 06:58:50.354961 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2448,"thread_start_us":65,"threads_started":1}
I20260926 06:58:50.355736 12714 heartbeater.cc:347] Connected to a master server at 127.1.74.126:46521
I20260926 06:58:50.355842 12714 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:50.356033 12714 heartbeater.cc:511] Master 127.1.74.126:46521 requested a full tablet report, sending...
I20260926 06:58:50.356673 12504 ts_manager.cc:194] Registered new tserver with Master: 2f1be6219bbc48429ac64d9181d3fefb (127.1.74.65:37155)
I20260926 06:58:50.357625 12504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:35051
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:58:50.378489 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:52789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46521
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:58:50.489379 12722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:50.489605 12722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:50.489643 12722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:50.490767 12722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:50.490837 12722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:50.490886 12722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:58:50.492378 12722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.74.84:52789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:50.492636 12722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:50.492859 12722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:50.498360 12730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:50.498646 12732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:50.498337 12729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:50.498775 12722 server_base.cc:1034] running on GCE node
I20260926 06:58:50.499092 12722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:50.499393 12722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:50.500550 12722 hybrid_clock.cc:648] HybridClock initialized: now 1790405930500541 us; error 29 us; skew 500 ppm
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:58:50.503072 12722 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:58:50.503551 12722 webserver.cc:460] Webserver started at http://127.1.74.66:42351/ using document root <none> and password file <none>
I20260926 06:58:50.503796 12722 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:50.503863 12722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:50.504004 12722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:58:50.514957 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "881599722b554012a8924fee568b9f0d"
format_stamp: "Formatted at 2026-09-26 06:58:50 on dist-test-slave-sfvt"
server_key: "fa7a17ef0ad02afdc1a09f625d070e08"
server_key_iv: "89edf49b6f34bc07d6becdf6d7d7c749"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:50.515406 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "881599722b554012a8924fee568b9f0d"
format_stamp: "Formatted at 2026-09-26 06:58:50 on dist-test-slave-sfvt"
server_key: "fa7a17ef0ad02afdc1a09f625d070e08"
server_key_iv: "89edf49b6f34bc07d6becdf6d7d7c749"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:50.519635 12722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:58:50.534232 12739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:50.534803 12722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:58:50.534914 12722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "881599722b554012a8924fee568b9f0d"
format_stamp: "Formatted at 2026-09-26 06:58:50 on dist-test-slave-sfvt"
server_key: "fa7a17ef0ad02afdc1a09f625d070e08"
server_key_iv: "89edf49b6f34bc07d6becdf6d7d7c749"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:50.535033 12722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:50.548612 12722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:50.551031 12722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:50.551178 12722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:50.551414 12722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:50.551777 12722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:50.551829 12722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:50.551867 12722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:50.551889 12722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:50.558493 12722 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:39997
I20260926 06:58:50.558903 12722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:58:50.560714 12852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:39997 every 8 connection(s)
I20260926 06:58:50.566118 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 12722
I20260926 06:58:50.566223 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:58:50.566404 1321 external_mini_cluster.cc:1428] Setting key d0503dc520fa00d7eb8ab548772d2422
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:58:50.581187 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.561822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:53537 (local address 127.1.74.126:46521)
0926 06:58:50.561877 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:50.561879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:50.568554 (+ 6675us) server_negotiation.cc:407] Connection header received
0926 06:58:50.569290 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:50.569291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:50.569303 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:50.569334 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:50.570197 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.571004 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.572528 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.572827 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.572888 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.578085 (+ 5197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:50.578086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:50.578087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:50.578094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:50.579805 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:50.580256 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:50.580257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:50.580257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:50.580305 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:50.580525 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:50.580526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:50.580526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:50.580642 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:50.580687 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:50.581009 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 06:58:50.581090 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:58:50.581902 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.561539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:46521 (local address 127.1.74.66:53537)
0926 06:58:50.568407 (+ 6868us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:50.568462 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:50.569161 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:50.569534 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:50.569540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:50.569784 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:50.570075 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.570083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.571125 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.571126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.572424 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.572427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.573021 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.573022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.573164 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.576283 (+ 3119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:50.576294 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:50.577976 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:50.579898 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:50.579904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:50.579907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:50.580184 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:50.580361 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:50.580361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:50.580361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:50.580466 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:50.580713 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:50.580717 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:50.580877 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 06:58:50.581520 (+ 643us) client_negotiation.cc:240] Negotiation successful
0926 06:58:50.581667 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6818,"thread_start_us":85,"threads_started":1}
I20260926 06:58:50.582463 12853 heartbeater.cc:347] Connected to a master server at 127.1.74.126:46521
I20260926 06:58:50.582543 12853 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:50.582705 12853 heartbeater.cc:511] Master 127.1.74.126:46521 requested a full tablet report, sending...
I20260926 06:58:50.583001 12504 ts_manager.cc:194] Registered new tserver with Master: 881599722b554012a8924fee568b9f0d (127.1.74.66:39997)
I20260926 06:58:50.583498 12504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:53537
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:58:50.610414 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:52789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46521
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:33821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:58:50.732143 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:58:50.732486 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:58:50.732569 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:58:50.734299 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:58:50.734409 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:58:50.734498 12858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:58:50.736811 12858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:33821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.74.84:52789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:58:50.737267 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:58:50.737640 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:58:50.744128 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:50.744212 12865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:58:50.744382 12864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:58:50.744518 12858 server_base.cc:1034] running on GCE node
I20260926 06:58:50.744717 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:58:50.744987 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:58:50.746158 12858 hybrid_clock.cc:648] HybridClock initialized: now 1790405930746131 us; error 42 us; skew 500 ppm
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:58:50.749468 12858 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:58:50.750069 12858 webserver.cc:460] Webserver started at http://127.1.74.67:34085/ using document root <none> and password file <none>
I20260926 06:58:50.750376 12858 fs_manager.cc:359] Metadata directory not provided
I20260926 06:58:50.750489 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:58:50.750671 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:58:50.771778 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "760e9a1e7b674bb3a8779b2f2857cefc"
format_stamp: "Formatted at 2026-09-26 06:58:50 on dist-test-slave-sfvt"
server_key: "dd238456ad1b5aa3f17c3292eded2fd7"
server_key_iv: "de92ecbbc07089287c3b216bd58e5e0a"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:50.772236 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "760e9a1e7b674bb3a8779b2f2857cefc"
format_stamp: "Formatted at 2026-09-26 06:58:50 on dist-test-slave-sfvt"
server_key: "dd238456ad1b5aa3f17c3292eded2fd7"
server_key_iv: "de92ecbbc07089287c3b216bd58e5e0a"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:50.774026 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 06:58:50.793874 12874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:50.794200 12858 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260926 06:58:50.794371 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "760e9a1e7b674bb3a8779b2f2857cefc"
format_stamp: "Formatted at 2026-09-26 06:58:50 on dist-test-slave-sfvt"
server_key: "dd238456ad1b5aa3f17c3292eded2fd7"
server_key_iv: "de92ecbbc07089287c3b216bd58e5e0a"
server_key_version: "kuduclusterkey@0"
I20260926 06:58:50.794549 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:58:50.819196 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:58:50.821669 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:58:50.821817 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:58:50.822080 12858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:58:50.822465 12858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:58:50.822515 12858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:50.822552 12858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:58:50.822578 12858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:58:50.829993 12858 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:46213
I20260926 06:58:50.830371 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:58:50.830858 12987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:46213 every 8 connection(s)
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:58:50.837481 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 12858
I20260926 06:58:50.837589 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:58:50.837837 1321 external_mini_cluster.cc:1428] Setting key f709ae7c87317089db5618b8c7c705fd
I20260926 06:58:50.840464 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.831159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:45055 (local address 127.1.74.126:46521)
0926 06:58:50.831233 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:50.831236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:50.831464 (+ 228us) server_negotiation.cc:407] Connection header received
0926 06:58:50.832076 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:50.832077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:50.832087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:50.832132 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:50.832803 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.833270 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.834668 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.834849 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.834903 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.837171 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:50.837172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:50.837173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:50.837181 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:50.838933 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:50.839488 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:50.839489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:50.839490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:50.839546 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:50.839807 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:50.839808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:50.839808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:50.839928 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:50.840002 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:50.840294 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 06:58:50.840370 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:58:50.840729 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.831118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:46521 (local address 127.1.74.67:45055)
0926 06:58:50.831315 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:50.831362 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:50.831964 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:50.832209 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:50.832213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:50.832450 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:58:50.832719 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.832726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.833360 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.833361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.834581 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.834583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.834932 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.834933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.835087 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.835623 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:58:50.835634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:58:50.837078 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:58:50.839005 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:50.839015 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:50.839017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:50.839400 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:50.839620 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:58:50.839621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:58:50.839621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:58:50.839735 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:58:50.839993 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:58:50.839998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:58:50.840150 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 06:58:50.840262 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 06:58:50.840402 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260926 06:58:50.841225 12988 heartbeater.cc:347] Connected to a master server at 127.1.74.126:46521
I20260926 06:58:50.841305 12988 heartbeater.cc:464] Registering TS with master...
I20260926 06:58:50.841502 12988 heartbeater.cc:511] Master 127.1.74.126:46521 requested a full tablet report, sending...
I20260926 06:58:50.841871 12504 ts_manager.cc:194] Registered new tserver with Master: 760e9a1e7b674bb3a8779b2f2857cefc (127.1.74.67:46213)
I20260926 06:58:50.842242 12504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:45055
I20260926 06:58:50.850040 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:58:50.869280 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.852446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.1.74.126:46521)
0926 06:58:50.852504 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:50.852507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:50.852604 (+ 97us) server_negotiation.cc:407] Connection header received
0926 06:58:50.852676 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:50.852677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:50.852686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:50.852730 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:58:50.853976 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.854460 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.856089 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.856265 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.856319 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.857844 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:58:50.857845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:58:50.857845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:58:50.857852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:58:50.866918 (+ 9066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:50.868345 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:50.868347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:50.868347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:50.868411 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:58:50.868624 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:58:50.868625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:58:50.868625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:58:50.868760 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:58:50.868836 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:50.869094 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 06:58:50.869181 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:58:50.871467 12504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:58:50.873739 12504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:58:50.882000 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.876819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:37155 (local address 127.0.0.1:55650)
0926 06:58:50.877541 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:50.877551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:50.877595 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:50.877900 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:50.877900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:50.877903 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:50.877953 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.877955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.879305 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.879306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.880644 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.880647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.881693 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.881694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.881774 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.881788 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:58:50.881856 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 06:58:50.881905 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":72,"threads_started":1}
I20260926 06:58:50.882236 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.876907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.1.74.65:37155)
0926 06:58:50.877696 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:50.877701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:50.877710 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:58:50.877742 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:50.877744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:50.877757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:50.877828 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:50.878047 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.879173 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.880748 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.881526 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.881598 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.881658 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:50.882127 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 06:58:50.882167 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":57,"threads_started":1}
I20260926 06:58:50.883111 12632 tablet_service.cc:1467] Processing CreateTablet for tablet 70c2848bdce044d0a89bc4c8116ea29b (DEFAULT_TABLE table=test-table [id=382e60ba241341cdac359b5fff13ecc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:58:50.883507 12632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c2848bdce044d0a89bc4c8116ea29b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:50.886986 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.876238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:39997 (local address 127.0.0.1:45072)
0926 06:58:50.877071 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:50.877083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:50.877149 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:50.883288 (+ 6139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:50.883289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:50.883292 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:50.883328 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.883329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.884664 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.884665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.885739 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.885741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.886760 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.886761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.886825 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.886830 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:58:50.886861 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 06:58:50.886897 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":77,"threads_started":1}
I20260926 06:58:50.887341 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.876599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.1.74.66:39997)
0926 06:58:50.876738 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:50.876743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:50.883081 (+ 6338us) server_negotiation.cc:407] Connection header received
0926 06:58:50.883136 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:50.883138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:50.883154 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:50.883406 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:50.883487 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.884561 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.885868 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.886657 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.887091 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.887179 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:50.887206 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 06:58:50.887243 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260926 06:58:50.887524 13006 tablet_bootstrap.cc:492] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb: Bootstrap starting.
I20260926 06:58:50.888049 12787 tablet_service.cc:1467] Processing CreateTablet for tablet 70c2848bdce044d0a89bc4c8116ea29b (DEFAULT_TABLE table=test-table [id=382e60ba241341cdac359b5fff13ecc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:58:50.888396 12787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c2848bdce044d0a89bc4c8116ea29b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:50.888435 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.876486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.1.74.67:46213)
0926 06:58:50.877281 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:50.877286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:50.877295 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:58:50.877333 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:50.877335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:50.877350 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:50.877434 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:50.877911 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.879094 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.880838 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.881595 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.881936 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.882020 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:50.888304 (+ 6284us) server_negotiation.cc:299] Negotiation successful
0926 06:58:50.888354 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":91,"threads_started":1}
I20260926 06:58:50.888206 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:50.876202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:46213 (local address 127.0.0.1:42444)
0926 06:58:50.876988 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:50.877029 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:50.877105 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:50.877742 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:50.877746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:50.877748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:50.877824 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.877826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.879191 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.879192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.880719 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:50.880722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:50.887935 (+ 7213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:50.887938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:50.888020 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:50.888024 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:58:50.888094 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 06:58:50.888134 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":83,"threads_started":1}
I20260926 06:58:50.889474 12912 tablet_service.cc:1467] Processing CreateTablet for tablet 70c2848bdce044d0a89bc4c8116ea29b (DEFAULT_TABLE table=test-table [id=382e60ba241341cdac359b5fff13ecc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:58:50.889778 12912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c2848bdce044d0a89bc4c8116ea29b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:58:50.892328 13006 tablet_bootstrap.cc:654] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:50.893088 13006 log.cc:824] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:50.893407 13008 tablet_bootstrap.cc:492] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc: Bootstrap starting.
I20260926 06:58:50.894413 13008 tablet_bootstrap.cc:654] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:50.894688 13008 log.cc:824] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:50.894985 13007 tablet_bootstrap.cc:492] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d: Bootstrap starting.
I20260926 06:58:50.895900 13007 tablet_bootstrap.cc:654] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d: Neither blocks nor log segments found. Creating new log.
I20260926 06:58:50.896191 13007 log.cc:824] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:58:50.896935 13006 tablet_bootstrap.cc:492] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb: No bootstrap required, opened a new log
I20260926 06:58:50.897008 13006 ts_tablet_manager.cc:1397] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260926 06:58:50.899639 13006 raft_consensus.cc:348] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } }
I20260926 06:58:50.899806 13006 raft_consensus.cc:374] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:50.899838 13006 raft_consensus.cc:729] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1be6219bbc48429ac64d9181d3fefb, State: Initialized, Role: FOLLOWER
I20260926 06:58:50.899986 13006 consensus_queue.cc:260] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } }
I20260926 06:58:50.900249 13006 ts_tablet_manager.cc:1428] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 06:58:50.900358 12714 heartbeater.cc:503] Master 127.1.74.126:46521 was elected leader, sending a full tablet report...
I20260926 06:58:50.902390 13008 tablet_bootstrap.cc:492] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc: No bootstrap required, opened a new log
I20260926 06:58:50.902459 13008 ts_tablet_manager.cc:1397] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260926 06:58:50.904500 13008 raft_consensus.cc:348] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } }
I20260926 06:58:50.904672 13008 raft_consensus.cc:374] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:50.904708 13008 raft_consensus.cc:729] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 760e9a1e7b674bb3a8779b2f2857cefc, State: Initialized, Role: FOLLOWER
I20260926 06:58:50.904851 13008 consensus_queue.cc:260] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } }
I20260926 06:58:50.905124 13008 ts_tablet_manager.cc:1428] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:58:50.905237 12988 heartbeater.cc:503] Master 127.1.74.126:46521 was elected leader, sending a full tablet report...
I20260926 06:58:50.906448 13007 tablet_bootstrap.cc:492] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d: No bootstrap required, opened a new log
I20260926 06:58:50.906533 13007 ts_tablet_manager.cc:1397] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260926 06:58:50.908603 13007 raft_consensus.cc:348] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } }
I20260926 06:58:50.908763 13007 raft_consensus.cc:374] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:58:50.908799 13007 raft_consensus.cc:729] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881599722b554012a8924fee568b9f0d, State: Initialized, Role: FOLLOWER
I20260926 06:58:50.908955 13007 consensus_queue.cc:260] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } }
I20260926 06:58:50.909258 13007 ts_tablet_manager.cc:1428] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:58:50.909428 12853 heartbeater.cc:503] Master 127.1.74.126:46521 was elected leader, sending a full tablet report...
Sep 26 06:58:50 dist-test-slave-sfvt krb5kdc[12028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405930, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
W20260926 06:58:51.061277 12854 tablet.cc:2369] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:58:51.081019 12989 tablet.cc:2369] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:58:51.094197 12715 tablet.cc:2369] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:58:51.099884 13012 raft_consensus.cc:482] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:58:51.100023 13012 raft_consensus.cc:504] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } }
I20260926 06:58:51.100419 13012 leader_election.cc:288] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 760e9a1e7b674bb3a8779b2f2857cefc (127.1.74.67:46213), 881599722b554012a8924fee568b9f0d (127.1.74.66:39997)
I20260926 06:58:51.104600 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:51.100827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:56529 (local address 127.1.74.67:46213)
0926 06:58:51.100891 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:51.100893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:51.101010 (+ 117us) server_negotiation.cc:407] Connection header received
0926 06:58:51.101124 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:51.101125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:51.101134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:51.101163 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:51.101375 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.102108 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.103669 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.104217 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.104265 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:51.104274 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:51.104489 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 06:58:51.104529 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:58:51.104617 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:51.100800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:52225 (local address 127.1.74.66:39997)
0926 06:58:51.100851 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:51.100853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:51.100976 (+ 123us) server_negotiation.cc:407] Connection header received
0926 06:58:51.101080 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:51.101081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:51.101093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:51.101122 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:58:51.101359 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.102061 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.103645 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.104216 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.104263 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:51.104273 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:51.104499 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 06:58:51.104538 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:58:51.104857 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:51.100618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:39997 (local address 127.1.74.65:52225)
0926 06:58:51.100903 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:51.100915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:51.101004 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:51.101187 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:51.101188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:51.101190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:51.101277 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:51.101279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.102161 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.102162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:51.103531 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:51.103534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.104332 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.104333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:51.104416 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:51.104421 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:58:51.104724 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 06:58:51.104766 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":90,"thread_start_us":83,"threads_started":1}
I20260926 06:58:51.105244 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:51.100619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:46213 (local address 127.1.74.65:56529)
0926 06:58:51.100943 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:51.100954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:58:51.101036 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:58:51.101226 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:58:51.101227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:58:51.101228 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:58:51.101306 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:51.101308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.102227 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.102228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:51.103568 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:58:51.103571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.104333 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.104334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:58:51.104411 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:51.104415 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:58:51.105121 (+ 706us) client_negotiation.cc:240] Negotiation successful
0926 06:58:51.105161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":12,"thread_start_us":105,"threads_started":1}
I20260926 06:58:51.105327 12788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c2848bdce044d0a89bc4c8116ea29b" candidate_uuid: "2f1be6219bbc48429ac64d9181d3fefb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881599722b554012a8924fee568b9f0d" is_pre_election: true
I20260926 06:58:51.105505 12788 raft_consensus.cc:2454] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1be6219bbc48429ac64d9181d3fefb in term 0.
I20260926 06:58:51.105522 12942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c2848bdce044d0a89bc4c8116ea29b" candidate_uuid: "2f1be6219bbc48429ac64d9181d3fefb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" is_pre_election: true
I20260926 06:58:51.105654 12942 raft_consensus.cc:2454] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f1be6219bbc48429ac64d9181d3fefb in term 0.
I20260926 06:58:51.105779 12601 leader_election.cc:302] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1be6219bbc48429ac64d9181d3fefb, 881599722b554012a8924fee568b9f0d; no voters:
I20260926 06:58:51.105913 13012 raft_consensus.cc:2781] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:58:51.105966 13012 raft_consensus.cc:482] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:58:51.105988 13012 raft_consensus.cc:3037] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:58:51.107522 13012 raft_consensus.cc:504] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } }
I20260926 06:58:51.107689 13012 leader_election.cc:288] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [CANDIDATE]: Term 1 election: Requested vote from peers 760e9a1e7b674bb3a8779b2f2857cefc (127.1.74.67:46213), 881599722b554012a8924fee568b9f0d (127.1.74.66:39997)
I20260926 06:58:51.107836 12942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c2848bdce044d0a89bc4c8116ea29b" candidate_uuid: "2f1be6219bbc48429ac64d9181d3fefb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "760e9a1e7b674bb3a8779b2f2857cefc"
I20260926 06:58:51.107916 12942 raft_consensus.cc:3037] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:58:51.107946 12788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c2848bdce044d0a89bc4c8116ea29b" candidate_uuid: "2f1be6219bbc48429ac64d9181d3fefb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881599722b554012a8924fee568b9f0d"
I20260926 06:58:51.108032 12788 raft_consensus.cc:3037] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:58:51.108877 12942 raft_consensus.cc:2454] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f1be6219bbc48429ac64d9181d3fefb in term 1.
I20260926 06:58:51.109092 12603 leader_election.cc:302] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f1be6219bbc48429ac64d9181d3fefb, 760e9a1e7b674bb3a8779b2f2857cefc; no voters:
I20260926 06:58:51.109253 13012 raft_consensus.cc:2781] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:58:51.109391 12788 raft_consensus.cc:2454] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f1be6219bbc48429ac64d9181d3fefb in term 1.
I20260926 06:58:51.109417 13012 raft_consensus.cc:686] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [term 1 LEADER]: Becoming Leader. State: Replica: 2f1be6219bbc48429ac64d9181d3fefb, State: Running, Role: LEADER
I20260926 06:58:51.109509 13012 consensus_queue.cc:237] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } }
I20260926 06:58:51.110294 12501 catalog_manager.cc:5495] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f1be6219bbc48429ac64d9181d3fefb (127.1.74.65). New cstate: current_term: 1 leader_uuid: "2f1be6219bbc48429ac64d9181d3fefb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f1be6219bbc48429ac64d9181d3fefb" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 } health_report { overall_health: UNKNOWN } } }
I20260926 06:58:51.122470 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:51.119191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.1.74.65:37155)
0926 06:58:51.119239 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:58:51.119241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:58:51.119279 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:58:51.119318 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:58:51.119319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:58:51.119339 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:58:51.119417 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:58:51.119571 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.120255 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.121374 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:58:51.121682 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:58:51.121746 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:58:51.121934 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:58:51.122060 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:58:51.122130 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:58:51.122341 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 06:58:51.122358 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:58:51.129336 12788 raft_consensus.cc:1261] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [term 1 FOLLOWER]: Refusing update from remote peer 2f1be6219bbc48429ac64d9181d3fefb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:58:51.129741 13012 consensus_queue.cc:1035] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [LEADER]: Connected to new peer: Peer: permanent_uuid: "881599722b554012a8924fee568b9f0d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:58:51.131839 12942 raft_consensus.cc:1261] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [term 1 FOLLOWER]: Refusing update from remote peer 2f1be6219bbc48429ac64d9181d3fefb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:58:51.132153 13012 consensus_queue.cc:1035] T 70c2848bdce044d0a89bc4c8116ea29b P 2f1be6219bbc48429ac64d9181d3fefb [LEADER]: Connected to new peer: Peer: permanent_uuid: "760e9a1e7b674bb3a8779b2f2857cefc" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:58:51.135198 13021 mvcc.cc:205] Tried to move back new op lower bound from 7333502693898522624 to 7333502693824892928. Current Snapshot: MvccSnapshot[applied={T|T < 7333502693898522624}]
I20260926 06:58:51.135622 13020 mvcc.cc:205] Tried to move back new op lower bound from 7333502693898522624 to 7333502693824892928. Current Snapshot: MvccSnapshot[applied={T|T < 7333502693898522624}]
I20260926 06:58:51.137569 13023 mvcc.cc:205] Tried to move back new op lower bound from 7333502693898522624 to 7333502693824892928. Current Snapshot: MvccSnapshot[applied={T|T < 7333502693898522624}]
I20260926 06:58:51.140678 12504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:58:51.140776 12504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:58:51.141450 12504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76d033b8a7b543f9aaae05abb3249a01: Sending DeleteTablet for 3 replicas of tablet 70c2848bdce044d0a89bc4c8116ea29b
I20260926 06:58:51.141881 12912 tablet_service.cc:1514] Processing DeleteTablet for tablet 70c2848bdce044d0a89bc4c8116ea29b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:58:51 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:42444
I20260926 06:58:51.142220 12632 tablet_service.cc:1514] Processing DeleteTablet for tablet 70c2848bdce044d0a89bc4c8116ea29b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:58:51 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:55650
I20260926 06:58:51.142467 12787 tablet_service.cc:1514] Processing DeleteTablet for tablet 70c2848bdce044d0a89bc4c8116ea29b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:58:51 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:45072
I20260926 06:58:51.142995 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 12585
I20260926 06:58:51.147415 13031 tablet_replica.cc:330] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d: stopping tablet replica
I20260926 06:58:51.151448 13029 tablet_replica.cc:330] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc: stopping tablet replica
I20260926 06:58:51.151654 13029 raft_consensus.cc:2229] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:58:51.151793 13031 raft_consensus.cc:2229] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:58:51.151978 13031 raft_consensus.cc:2258] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:58:51.152119 13029 raft_consensus.cc:2258] T 70c2848bdce044d0a89bc4c8116ea29b P 760e9a1e7b674bb3a8779b2f2857cefc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 06:58:51.152578 12489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:37155 (error 108)
I20260926 06:58:51.152834 13031 ts_tablet_manager.cc:1905] T 70c2848bdce044d0a89bc4c8116ea29b P 881599722b554012a8924fee568b9f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:58:51.153290 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 12722
I20260926 06:58:51.159667 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:51.159533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:37155 (local address 127.0.0.1:55660)
0926 06:58:51.159586 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:51.159623 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:37155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 06:58:51.160017 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:58:51.159937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:39997 (local address 127.0.0.1:45088)
0926 06:58:51.159973 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 06:58:51.159993 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.66:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 06:58:51.160207 12489 catalog_manager.cc:4584] TS 2f1be6219bbc48429ac64d9181d3fefb (127.1.74.65:37155): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70c2848bdce044d0a89bc4c8116ea29b: Network error: Client connection negotiation failed: client connection to 127.1.74.65:37155: connect: Connection refused (error 111)
I20260926 06:58:51.160636 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 12858
I20260926 06:58:51.165846 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 12472
I20260926 06:58:51.172860 1321 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 06:58:51.518687 1321 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 06:58:51.518805 1321 mini_ranger.cc:66] Stopping Ranger...
I20260926 06:58:51.863349 1321 mini_ranger.cc:68] Stopped Ranger
2026-09-26 06:58:51.863 UTC [12058] LOG: received smart shutdown request
2026-09-26 06:58:51.866 UTC [12058] LOG: background worker "logical replication launcher" (PID 12067) exited with exit code 1
2026-09-26 06:58:51.866 UTC [12062] LOG: shutting down
2026-09-26 06:58:52.040 UTC [12058] LOG: database system is shut down
2026-09-26T06:58:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44928 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:58:52 dist-test-slave-sfvt krb5kdc[13041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:58:52 dist-test-slave-sfvt krb5kdc[13041](info): set up 2 sockets
Sep 26 06:58:52 dist-test-slave-sfvt krb5kdc[13041](info): commencing operation
krb5kdc: starting...
W20260926 06:58:53.183144 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:58:53 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:58:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.74.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.74.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/rangeradmin_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/rangeradmin_127.1.74.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.74.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.74.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/rangerlookup_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/rangerlookup_127.1.74.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.74.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.74.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/HTTP_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/HTTP_127.1.74.84@KRBTEST.COM.keytab.
I20260926 06:58:53.350405 1321 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taske4hwn4/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/postgres -l logfile start
2026-09-26 06:58:56.179 UTC [13071] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 06:58:56.179 UTC [13071] LOG: listening on IPv4 address "127.1.74.84", port 60115
2026-09-26 06:58:56.182 UTC [13071] LOG: listening on Unix socket "/tmp/.s.PGSQL.60115"
2026-09-26 06:58:56.209 UTC [13074] LOG: database system was shut down at 2026-09-26 06:58:54 UTC
2026-09-26 06:58:56.213 UTC [13071] LOG: database system is ready to accept connections
I20260926 06:58:57.159340 1321 mini_postgres.cc:96] Postgres bound to 60115
2026-09-26 06:58:57.165 UTC [13082] FATAL: database "slave" does not exist
127.1.74.84:60115 - accepting connections
I20260926 06:58:57.165596 1321 mini_ranger.cc:161] Starting Ranger...
I20260926 06:58:57.180435 1321 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 06:58:57.662581 1321 mini_ranger.cc:87] Created ranger Postgres database
I20260926 06:58:57.662678 1321 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-admin
2026-09-26 06:58:57,968 [I] DB FLAVOR :POSTGRES
2026-09-26 06:58:57,968 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:58:57,968 [I] Checking connection..
2026-09-26 06:58:57,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 06:58:58,197 [I] Checking connection passed.
2026-09-26 06:58:58,198 [I] --------- Verifying version history table ---------
2026-09-26 06:58:58,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:58:58,452 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:58:58,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:58:58,699 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:58:58,699 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 06:58:58,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 06:58:58.931 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:58:58,937 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 06:58:58,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:58:59,199 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 06:58:59,200 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:58:59,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 06:58:59,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 06:58:59,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-sfvt.c.gcp-upstream.internal','N') ;"
2026-09-26 06:58:59,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:59:00,139 [I] Table x_portal_user does not exist in database ranger
2026-09-26 06:59:00,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:59:00,405 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 06:59:00,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:59:00,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:59:00,884 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 06:59:00,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 06:59:01.842 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:01.858 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:01.875 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:01.890 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:01.910 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:01.926 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:01.983 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:01.990 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:02.004 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:02.014 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:02.024 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:02.038 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:02.048 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:02.054 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:02.061 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:02.068 UTC [13254] WARNING: there is no transaction in progress
2026-09-26 06:59:02,720 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 06:59:02,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 06:59:02,949 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 06:59:02,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:59:03,217 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 06:59:03,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:59:03,455 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 06:59:03,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:59:03,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:59:03,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-sfvt.c.gcp-upstream.internal';"
2026-09-26 06:59:04,133 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 06:59:04,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 06:59:04,358 [I] DB_PATCHES have already been applied
I20260926 06:59:04.363626 1321 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-admin:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taske4hwn4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 06:59:04.363684 1321 mini_ranger.cc:193] Using host: 127.1.74.84
I20260926 06:59:04.366624 1321 mini_ranger.cc:227] Ranger admin URL: http://127.1.74.84:60219
Sep 26, 2026 6:59:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:59:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:59:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 6:59:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 6:59:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 6:59:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 6:59:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 06:59:05.479748 1321 mini_ranger.cc:160] Time spent starting Ranger: real 8.314s user 0.000s sys 0.008s
Sep 26, 2026 6:59:25 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 6:59:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 26, 2026 6:59:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 6:59:25 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 6:59:26 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 06:59:26.711988 1321 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.74.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.74.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/rangerkms_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.74.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/rangerkms_127.1.74.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.74.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.74.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.74.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/HTTP_127.1.74.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.74.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/HTTP_127.1.74.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 06:59:26 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405966, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 06:59:26.791131 1321 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 06:59:26.800643 1321 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 06:59:27.703550 1321 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 06:59:27.703634 1321 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-kms
I20260926 06:59:27.703652 1321 mini_ranger_kms.cc:227] Using postgres at 127.1.74.84:60115
2026-09-26 06:59:27,863 [I] DB FLAVOR :POSTGRES
2026-09-26 06:59:27,863 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:59:27,863 [I] Checking connection
2026-09-26 06:59:27,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 06:59:28,100 [I] connection success
2026-09-26 06:59:28,100 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 06:59:28,100 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 06:59:28,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 06:59:28,328 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 06:59:28,328 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:59:28,328 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 06:59:28,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.74.84:60115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 06:59:28,579 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 06:59:29.018978 1321 mini_ranger_kms.cc:310] Created kms service
I20260926 06:59:29.226248 1321 mini_ranger_kms.cc:326] Created kudu user
I20260926 06:59:29.272428 1321 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 06:59:29 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405966, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:29.686586 1321 mini_ranger_kms.cc:384] Added ranger policy
I20260926 06:59:29.686810 1321 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-kms:/tmp/dist-test-taske4hwn4/build/release/bin/postgresql.jar:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taske4hwn4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 06:59:29.686832 1321 mini_ranger_kms.cc:240] Using host: 127.1.74.84
I20260926 06:59:29.690203 1321 mini_ranger_kms.cc:276] Ranger KMS PID: 13434
I20260926 06:59:29.690279 1321 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.74.84:42615
Sep 26, 2026 6:59:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:59:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:59:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 6:59:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 6:59:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 6:59:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 06:59:31.907536 1321 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.116s user 0.000s sys 0.143s
I20260926 06:59:31.907624 1321 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 06:59:31.907670 1321 mini_ranger_kms.cc:401] 127.1.74.84:42615/kms/v1/keys
Sep 26 06:59:31 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790405971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.74.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 06:59:32 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.74.84@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
Sep 26 06:59:32 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.74.84@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
Sep 26 06:59:34 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:59:34.914065 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33803
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:33803
--ranger_config_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:42615/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:34.994956 13485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:34.995155 13485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:34.995177 13485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:34.996327 13485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:59:34.996371 13485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:34.996385 13485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:34.996398 13485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:59:34.996410 13485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:59:34.997923 13485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.74.84:42615/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:33803
--ranger_config_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:34.998153 13485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:34.998451 13485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:35.003844 13491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:35.003849 13490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:35.003895 13485 server_base.cc:1034] running on GCE node
W20260926 06:59:35.004019 13493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:35.004366 13485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:35.004709 13485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:35.005877 13485 hybrid_clock.cc:648] HybridClock initialized: now 1790405975005851 us; error 45 us; skew 500 ppm
Sep 26 06:59:35 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:35.008948 13485 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:59:35.009473 13485 webserver.cc:460] Webserver started at http://127.1.74.126:44287/ using document root <none> and password file <none>
I20260926 06:59:35.009714 13485 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:35.009773 13485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:35.009922 13485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:59:35 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:35.054227 13485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "84489a825a2e43aa9b1b037e7e4223f5"
format_stamp: "Formatted at 2026-09-26 06:59:35 on dist-test-slave-sfvt"
server_key: "227fa412c64e1e1844801710a3ab90e3"
server_key_iv: "f3d53b6afcb3e61708039ffe835349a4"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:35.054628 13485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "84489a825a2e43aa9b1b037e7e4223f5"
format_stamp: "Formatted at 2026-09-26 06:59:35 on dist-test-slave-sfvt"
server_key: "227fa412c64e1e1844801710a3ab90e3"
server_key_iv: "f3d53b6afcb3e61708039ffe835349a4"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:35.056192 13485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:59:35.067900 13500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:35.068143 13485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:59:35.068245 13485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "84489a825a2e43aa9b1b037e7e4223f5"
format_stamp: "Formatted at 2026-09-26 06:59:35 on dist-test-slave-sfvt"
server_key: "227fa412c64e1e1844801710a3ab90e3"
server_key_iv: "f3d53b6afcb3e61708039ffe835349a4"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:35.068362 13485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:35.083684 13485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:35.085716 13485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:35.085875 13485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:35.090620 13485 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:33803
I20260926 06:59:35.090657 13552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:33803 every 8 connection(s)
I20260926 06:59:35.090999 13485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:59:35.091777 13553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:35.094882 13553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Bootstrap starting.
I20260926 06:59:35.095634 13553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:35.095978 13553 log.cc:824] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:35.096735 13553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: No bootstrap required, opened a new log
I20260926 06:59:35.099660 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 13485
I20260926 06:59:35.099583 13553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } }
I20260926 06:59:35.099779 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:59:35.099798 13553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:35.099834 13553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84489a825a2e43aa9b1b037e7e4223f5, State: Initialized, Role: FOLLOWER
I20260926 06:59:35.099960 1321 external_mini_cluster.cc:1428] Setting key 08558e38ec6434326eaa3d3a8981bac9
I20260926 06:59:35.100011 13553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } }
I20260926 06:59:35.100087 13553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:59:35.100153 13553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:59:35.100227 13553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:35.101275 13553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } }
I20260926 06:59:35.101388 13553 leader_election.cc:302] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84489a825a2e43aa9b1b037e7e4223f5; no voters:
I20260926 06:59:35.101585 13553 leader_election.cc:288] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:59:35.101641 13558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:59:35.101881 13558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 1 LEADER]: Becoming Leader. State: Replica: 84489a825a2e43aa9b1b037e7e4223f5, State: Running, Role: LEADER
I20260926 06:59:35.101951 13553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:59:35.102059 13558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } }
I20260926 06:59:35.102409 13558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84489a825a2e43aa9b1b037e7e4223f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } } }
I20260926 06:59:35.102532 13558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:35.102694 13558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84489a825a2e43aa9b1b037e7e4223f5. Latest consensus state: current_term: 1 leader_uuid: "84489a825a2e43aa9b1b037e7e4223f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } } }
I20260926 06:59:35.102787 13558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:35.102928 13567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:59:35.103466 13567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:59:35.103745 13553 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 06:59:35.107373 13575 mvcc.cc:205] Tried to move back new op lower bound from 7333502874028826624 to 7333502874018533376. Current Snapshot: MvccSnapshot[applied={T|T < 7333502874028826624}]
Sep 26 06:59:35 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:35.111922 13567 catalog_manager.cc:1347] Generated new cluster ID: bf1e4fb059ac4b418e549821fcd41ec1
I20260926 06:59:35.111974 13567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:59:35.113584 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:35.100611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.1.74.126:33803)
0926 06:59:35.100772 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:35.100780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:35.100804 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:59:35.100895 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:35.100900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:35.100944 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:35.101098 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:35.101899 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:35.103346 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:35.104414 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:35.105010 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:35.105089 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:35.109356 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:35.109364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:35.109364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:35.109377 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:35.111451 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:35.111860 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:35.111863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:35.111864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:35.111977 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:35.112187 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:35.112188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:35.112189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:35.112462 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:35.112595 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:35.113008 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 06:59:35.113141 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260926 06:59:35.136193 13567 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:59:35.136948 13567 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:59:35.141135 13567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Generated new TSK 0
I20260926 06:59:35.141309 13567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:59:35.801952 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:42615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33803
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:35.909116 13598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:35.909340 13598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:35.909379 13598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:35.910522 13598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:35.910583 13598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:35.910624 13598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:59:35.912107 13598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.74.84:42615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:35.912353 13598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:35.912575 13598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:35.918478 13603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:35.918495 13604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:35.918697 13606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:35.918820 13598 server_base.cc:1034] running on GCE node
I20260926 06:59:35.919157 13598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:35.919432 13598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:35.920634 13598 hybrid_clock.cc:648] HybridClock initialized: now 1790405975920618 us; error 34 us; skew 500 ppm
Sep 26 06:59:35 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:35.923218 13598 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:59:35.923779 13598 webserver.cc:460] Webserver started at http://127.1.74.65:35265/ using document root <none> and password file <none>
I20260926 06:59:35.924046 13598 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:35.924114 13598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:35.924259 13598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:59:35 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:35.936872 13598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "7d25d06e07e046f2976f96e0105ecb74"
format_stamp: "Formatted at 2026-09-26 06:59:35 on dist-test-slave-sfvt"
server_key: "8a41ba52a4e56800f7f3cf462b937070"
server_key_iv: "f3a9f8d1a1422bf56840be54e2414951"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:35.937474 13598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "7d25d06e07e046f2976f96e0105ecb74"
format_stamp: "Formatted at 2026-09-26 06:59:35 on dist-test-slave-sfvt"
server_key: "8a41ba52a4e56800f7f3cf462b937070"
server_key_iv: "f3a9f8d1a1422bf56840be54e2414951"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:35.941534 13598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260926 06:59:35.956550 13613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:35.956832 13598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:35.956983 13598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "7d25d06e07e046f2976f96e0105ecb74"
format_stamp: "Formatted at 2026-09-26 06:59:35 on dist-test-slave-sfvt"
server_key: "8a41ba52a4e56800f7f3cf462b937070"
server_key_iv: "f3a9f8d1a1422bf56840be54e2414951"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:35.957149 13598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:35.976887 13598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:35.979176 13598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:35.979413 13598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:35.979683 13598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:35.980042 13598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:35.980079 13598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:35.980109 13598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:35.980129 13598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:35.987171 13598 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:45461
I20260926 06:59:35.987231 13727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:45461 every 8 connection(s)
I20260926 06:59:35.987624 13598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:59:35.988874 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 13598
I20260926 06:59:35.988974 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:59:35.989166 1321 external_mini_cluster.cc:1428] Setting key a06b90788ecf422addd9e56c01b95a5a
Sep 26 06:59:35 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
I20260926 06:59:36.001632 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:35.990256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:56283 (local address 127.1.74.126:33803)
0926 06:59:35.992052 (+ 1796us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:35.992054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:35.992062 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:59:35.992084 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:35.992085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:35.992097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:35.992905 (+ 808us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:35.992946 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:35.993765 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:35.995455 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:35.995817 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:35.996357 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:35.998484 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:35.998486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:35.998486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:35.998494 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:36.000149 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:36.000613 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:36.000614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:36.000615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:36.000677 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:36.000927 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:36.000928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:36.000929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:36.001054 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:36.001128 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:36.001427 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 06:59:36.001512 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1764,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
I20260926 06:59:36.002015 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:35.989683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33803 (local address 127.1.74.65:56283)
0926 06:59:35.990335 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:35.991380 (+ 1045us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:35.991844 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:35.992295 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:35.992300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:35.992537 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:35.992810 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:35.992815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:35.993872 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:35.993873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:35.995343 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:35.995347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:35.995940 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:35.995941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:35.996100 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:35.996710 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:35.996720 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:35.998368 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:36.000252 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:36.000258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:36.000261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:36.000525 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:36.000743 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:36.000743 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:36.000744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:36.000854 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:36.001129 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:36.001136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:36.001295 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 06:59:36.001408 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 06:59:36.001624 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":101,"threads_started":1}
I20260926 06:59:36.002547 13728 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33803
I20260926 06:59:36.002645 13728 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:36.002827 13728 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
I20260926 06:59:36.003491 13517 ts_manager.cc:194] Registered new tserver with Master: 7d25d06e07e046f2976f96e0105ecb74 (127.1.74.65:45461)
I20260926 06:59:36.004406 13517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:56283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:59:36.020277 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:42615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33803
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:36.124176 13735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:36.124392 13735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:36.124432 13735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:36.125779 13735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:36.125856 13735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:36.125901 13735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:59:36.127451 13735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.74.84:42615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:36.127708 13735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:36.127930 13735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:36.132858 13743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:36.132913 13745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:36.132856 13742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:36.133980 13735 server_base.cc:1034] running on GCE node
I20260926 06:59:36.134197 13735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:36.134490 13735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:36.135638 13735 hybrid_clock.cc:648] HybridClock initialized: now 1790405976135626 us; error 27 us; skew 500 ppm
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:36.138314 13735 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:59:36.138823 13735 webserver.cc:460] Webserver started at http://127.1.74.66:37929/ using document root <none> and password file <none>
I20260926 06:59:36.139065 13735 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:36.139135 13735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:36.139329 13735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:36.152030 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "c956c887663d4629b49a44cf56720ef1"
format_stamp: "Formatted at 2026-09-26 06:59:36 on dist-test-slave-sfvt"
server_key: "8c152674b896deb44468f271e1cb9d10"
server_key_iv: "9e5aa06cdfa7d803c0f3f8450dab04fe"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:36.152557 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "c956c887663d4629b49a44cf56720ef1"
format_stamp: "Formatted at 2026-09-26 06:59:36 on dist-test-slave-sfvt"
server_key: "8c152674b896deb44468f271e1cb9d10"
server_key_iv: "9e5aa06cdfa7d803c0f3f8450dab04fe"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:36.155764 13735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 06:59:36.176222 13752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.182034 13735 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.000s sys 0.000s
I20260926 06:59:36.182189 13735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "c956c887663d4629b49a44cf56720ef1"
format_stamp: "Formatted at 2026-09-26 06:59:36 on dist-test-slave-sfvt"
server_key: "8c152674b896deb44468f271e1cb9d10"
server_key_iv: "9e5aa06cdfa7d803c0f3f8450dab04fe"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:36.182309 13735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:36.203671 13735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:36.206271 13735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:36.206426 13735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:36.206683 13735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:36.207050 13735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:36.207108 13735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.207150 13735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:36.207171 13735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.214756 13735 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:41495
I20260926 06:59:36.215037 13865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:41495 every 8 connection(s)
I20260926 06:59:36.215226 13735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:59:36.216451 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 13735
I20260926 06:59:36.216585 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:59:36.216807 1321 external_mini_cluster.cc:1428] Setting key a63f0c5e92bcf49e6e42d85bcbe1b73a
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:36.228405 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:36.216468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33803 (local address 127.1.74.66:52845)
0926 06:59:36.216918 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:36.217208 (+ 290us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:36.218057 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:36.218436 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:36.218441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:36.218707 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:36.219001 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:36.219008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.219837 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.219838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:36.220641 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:36.220644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.221256 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.221258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:36.221433 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:36.222100 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:36.222114 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:36.224081 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:36.226427 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:36.226435 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:36.226438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:36.226789 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:36.227044 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:36.227045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:36.227046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:36.227209 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:36.227611 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:36.227619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:36.227827 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 06:59:36.227950 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 06:59:36.228124 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
I20260926 06:59:36.229019 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:36.216774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:52845 (local address 127.1.74.126:33803)
0926 06:59:36.216828 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:36.216830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:36.217294 (+ 464us) server_negotiation.cc:407] Connection header received
0926 06:59:36.218196 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:36.218198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:36.218214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:36.218249 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:36.219140 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.219736 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.220759 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.221095 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.221173 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:36.224205 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:36.224207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:36.224207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:36.224216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:36.226321 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:36.226894 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:36.226895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:36.226896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:36.226965 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:36.227316 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:36.227317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:36.227317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:36.227472 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:36.227552 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:36.228802 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0926 06:59:36.228899 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:59:36.229630 13866 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33803
I20260926 06:59:36.229746 13866 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:36.229964 13866 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
I20260926 06:59:36.230353 13517 ts_manager.cc:194] Registered new tserver with Master: c956c887663d4629b49a44cf56720ef1 (127.1.74.66:41495)
I20260926 06:59:36.230883 13517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:52845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:59:36.253558 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:42615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33803
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:36.344190 13871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:36.344486 13871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:36.344528 13871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:36.345734 13871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:36.345804 13871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:36.345851 13871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:59:36.347957 13871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.74.84:42615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:36.348186 13871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:36.348430 13871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:36.353720 13880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:36.353720 13878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:36.354068 13877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:36.354426 13871 server_base.cc:1034] running on GCE node
I20260926 06:59:36.354624 13871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:36.354885 13871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:36.356076 13871 hybrid_clock.cc:648] HybridClock initialized: now 1790405976356056 us; error 30 us; skew 500 ppm
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:36.359419 13871 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:59:36.359922 13871 webserver.cc:460] Webserver started at http://127.1.74.67:35977/ using document root <none> and password file <none>
I20260926 06:59:36.360212 13871 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:36.360265 13871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:36.360387 13871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:36.378062 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "98dce2412a41471c8197c8a2eda22ed6"
format_stamp: "Formatted at 2026-09-26 06:59:36 on dist-test-slave-sfvt"
server_key: "7f8b5788d103551c62f91fcb59acaa1b"
server_key_iv: "4cdb79e046d7343974b0c1e92161e04c"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:36.378510 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "98dce2412a41471c8197c8a2eda22ed6"
format_stamp: "Formatted at 2026-09-26 06:59:36 on dist-test-slave-sfvt"
server_key: "7f8b5788d103551c62f91fcb59acaa1b"
server_key_iv: "4cdb79e046d7343974b0c1e92161e04c"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:36.382727 13871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 06:59:36.399575 13887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.399863 13871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 06:59:36.399962 13871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "98dce2412a41471c8197c8a2eda22ed6"
format_stamp: "Formatted at 2026-09-26 06:59:36 on dist-test-slave-sfvt"
server_key: "7f8b5788d103551c62f91fcb59acaa1b"
server_key_iv: "4cdb79e046d7343974b0c1e92161e04c"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:36.400055 13871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:36.425649 13871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:36.428144 13871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:36.428303 13871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:36.428529 13871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:36.428917 13871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:36.428966 13871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.428997 13871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:36.429021 13871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.435674 13871 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:46095
I20260926 06:59:36.435832 14000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:46095 every 8 connection(s)
I20260926 06:59:36.436115 13871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:36.444103 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 13871
I20260926 06:59:36.444206 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:59:36.444393 1321 external_mini_cluster.cc:1428] Setting key 55a17da2fb297f3648d335e173868031
I20260926 06:59:36.447893 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:36.437172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:49715 (local address 127.1.74.126:33803)
0926 06:59:36.437223 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:36.437226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:36.437448 (+ 222us) server_negotiation.cc:407] Connection header received
0926 06:59:36.438230 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:36.438232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:36.438245 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:36.438280 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:36.439164 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.439982 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.441481 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.441776 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.441866 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:36.444577 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:36.444579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:36.444579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:36.444587 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:36.446324 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:36.446783 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:36.446784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:36.446784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:36.446844 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:36.447151 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:36.447152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:36.447153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:36.447340 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:36.447399 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:36.447709 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 06:59:36.447799 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:59:36.448096 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:36.436859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33803 (local address 127.1.74.67:49715)
0926 06:59:36.437309 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:36.437359 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:36.438073 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:36.438484 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:36.438490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:36.438754 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:36.439038 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:36.439045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.440114 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.440114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:36.441387 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:36.441390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.441952 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.441953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:36.442098 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:36.442767 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:36.442777 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:36.444461 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:36.446418 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:36.446425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:36.446428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:36.446689 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:36.446944 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:36.446945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:36.446945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:36.447070 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:36.447401 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:36.447408 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:36.447573 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 06:59:36.447690 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 06:59:36.447854 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":93,"threads_started":1}
I20260926 06:59:36.451773 14001 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33803
I20260926 06:59:36.451891 14001 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:36.452088 14001 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
I20260926 06:59:36.452502 13517 ts_manager.cc:194] Registered new tserver with Master: 98dce2412a41471c8197c8a2eda22ed6 (127.1.74.67:46095)
I20260926 06:59:36.452898 13517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:49715
I20260926 06:59:36.456220 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:59:36.456817 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 13598
I20260926 06:59:36.462255 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 13735
I20260926 06:59:36.466475 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 13871
I20260926 06:59:36.470958 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 13485
I20260926 06:59:36.476881 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33803
--webserver_interface=127.1.74.126
--webserver_port=44287
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:33803
--ranger_config_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:42615/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.74.84:42615/kms with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:36.558079 14004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:36.558310 14004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:36.558333 14004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:36.559581 14004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:59:36.559628 14004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:36.559643 14004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:36.559686 14004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:59:36.559717 14004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:59:36.561278 14004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.74.84:42615/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:33803
--ranger_config_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=44287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:36.561535 14004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:36.561758 14004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:36.566861 14012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:36.566934 14004 server_base.cc:1034] running on GCE node
W20260926 06:59:36.566825 14010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:36.566879 14009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:36.567327 14004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:36.567633 14004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:36.568797 14004 hybrid_clock.cc:648] HybridClock initialized: now 1790405976568785 us; error 29 us; skew 500 ppm
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:36.571174 14004 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:59:36.571645 14004 webserver.cc:460] Webserver started at http://127.1.74.126:44287/ using document root <none> and password file <none>
I20260926 06:59:36.571877 14004 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:36.571951 14004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:36.627233 14004 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:36.628157 14020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.628389 14004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:59:36.628490 14004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "84489a825a2e43aa9b1b037e7e4223f5"
format_stamp: "Formatted at 2026-09-26 06:59:35 on dist-test-slave-sfvt"
server_key: "227fa412c64e1e1844801710a3ab90e3"
server_key_iv: "f3d53b6afcb3e61708039ffe835349a4"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:36.628948 14004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:36.651485 14004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:36.653266 14004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:36.653381 14004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:36.657881 14004 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:33803
I20260926 06:59:36.657938 14072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:33803 every 8 connection(s)
I20260926 06:59:36.658303 14004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:59:36.659519 14073 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:59:36.660193 14073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Bootstrap starting.
I20260926 06:59:36.662797 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 14004
I20260926 06:59:36.663091 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:42615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:45461
--local_ip_for_outbound_sockets=127.1.74.65
--tserver_master_addrs=127.1.74.126:33803
--webserver_port=35265
--webserver_interface=127.1.74.65
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.74.84:42615/kms with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:36.668201 14073 log.cc:824] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:36.669874 14073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:59:36.670162 14073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Bootstrap complete.
I20260926 06:59:36.672581 14073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } }
I20260926 06:59:36.672840 14073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84489a825a2e43aa9b1b037e7e4223f5, State: Initialized, Role: FOLLOWER
I20260926 06:59:36.673020 14073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } }
I20260926 06:59:36.673099 14073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:59:36.673151 14073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:59:36.673197 14073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:59:36.674173 14073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } }
I20260926 06:59:36.674288 14073 leader_election.cc:302] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84489a825a2e43aa9b1b037e7e4223f5; no voters:
I20260926 06:59:36.674484 14073 leader_election.cc:288] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:59:36.674543 14077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:59:36.674683 14077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [term 2 LEADER]: Becoming Leader. State: Replica: 84489a825a2e43aa9b1b037e7e4223f5, State: Running, Role: LEADER
I20260926 06:59:36.674754 14073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:59:36.674783 14077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } }
I20260926 06:59:36.675040 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84489a825a2e43aa9b1b037e7e4223f5. Latest consensus state: current_term: 2 leader_uuid: "84489a825a2e43aa9b1b037e7e4223f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } } }
I20260926 06:59:36.675107 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:36.675043 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "84489a825a2e43aa9b1b037e7e4223f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84489a825a2e43aa9b1b037e7e4223f5" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33803 } } }
I20260926 06:59:36.675321 14082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:59:36.675325 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:36.675814 14082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:59:36.675938 14082 catalog_manager.cc:1259] Loaded cluster ID: bf1e4fb059ac4b418e549821fcd41ec1
I20260926 06:59:36.675971 14082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:59:36.676249 14073 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 06:59:36.679467 14082 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:59:36.680019 14082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Loaded TSK: 0
I20260926 06:59:36.680299 14082 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:59:36.750398 14075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:36.750656 14075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:36.750694 14075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:36.752419 14075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:36.752480 14075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:36.752521 14075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:59:36.754637 14075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.74.84:42615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:45461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=35265
--tserver_master_addrs=127.1.74.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:36.754916 14075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:36.755172 14075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:36.760545 14110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:36.760536 14113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:36.761000 14111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:36.761643 14075 server_base.cc:1034] running on GCE node
I20260926 06:59:36.761886 14075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:36.762140 14075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:36.765486 14075 hybrid_clock.cc:648] HybridClock initialized: now 1790405976765467 us; error 38 us; skew 500 ppm
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:36.768909 14075 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:59:36.769418 14075 webserver.cc:460] Webserver started at http://127.1.74.65:35265/ using document root <none> and password file <none>
I20260926 06:59:36.769680 14075 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:36.769762 14075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:36.796766 14075 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260926 06:59:36.803753 14121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.803949 14075 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260926 06:59:36.804045 14075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "7d25d06e07e046f2976f96e0105ecb74"
format_stamp: "Formatted at 2026-09-26 06:59:35 on dist-test-slave-sfvt"
server_key: "8a41ba52a4e56800f7f3cf462b937070"
server_key_iv: "f3a9f8d1a1422bf56840be54e2414951"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:36.804538 14075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:36.821990 14075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:36.824431 14075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:36.824586 14075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:36.824810 14075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:36.825181 14075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:36.825229 14075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.825260 14075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:36.825285 14075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:36.832963 14075 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:45461
I20260926 06:59:36.833395 14075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:59:36.836079 14234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:45461 every 8 connection(s)
I20260926 06:59:36.839962 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 14075
I20260926 06:59:36.840258 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:42615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:41495
--local_ip_for_outbound_sockets=127.1.74.66
--tserver_master_addrs=127.1.74.126:33803
--webserver_port=37929
--webserver_interface=127.1.74.66
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.74.84:42615/kms with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:36.849195 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:36.836722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33803 (local address 127.1.74.65:51145)
0926 06:59:36.836919 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:36.836969 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:36.837579 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:36.838263 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:36.838268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:36.838495 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:36.838776 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:36.838781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.840510 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.840511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:36.841754 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:36.841757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.842625 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.842626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:36.842781 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:36.843413 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:36.843424 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:36.845038 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:36.847385 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:36.847391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:36.847393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:36.847675 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:36.847964 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:36.847965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:36.847965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:36.848085 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:36.848541 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:36.848546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:36.848691 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 06:59:36.848811 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 06:59:36.848965 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260926 06:59:36.849964 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:36.837117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:51145 (local address 127.1.74.126:33803)
0926 06:59:36.837804 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:36.837813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:36.837848 (+ 35us) server_negotiation.cc:407] Connection header received
0926 06:59:36.838002 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:36.838009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:36.838062 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:36.838900 (+ 838us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:36.839001 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.840387 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.841864 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:36.842479 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:36.842597 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:36.845171 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:36.845175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:36.845176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:36.845189 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:36.847250 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:36.847782 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:36.847786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:36.847786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:36.847890 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:36.848155 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:36.848156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:36.848156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:36.848434 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:36.848512 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:36.849602 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0926 06:59:36.849718 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":84,"threads_started":1}
I20260926 06:59:36.851074 14235 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33803
I20260926 06:59:36.851162 14235 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:36.851364 14235 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
W20260926 06:59:36.852200 14235 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:33803 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 06:59:36.852262 14235 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:36.852314 14235 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
I20260926 06:59:36.852697 14235 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:36.852771 14235 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
W20260926 06:59:36.853055 14235 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:59:36.957209 14239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:36.957472 14239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:36.957513 14239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:36.958829 14239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:36.958884 14239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:36.958925 14239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:59:36.961148 14239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.74.84:42615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:41495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=37929
--tserver_master_addrs=127.1.74.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:36.961385 14239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:36.961653 14239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:36.968060 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:36.968319 14247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:36.969125 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:36.970616 14239 server_base.cc:1034] running on GCE node
I20260926 06:59:36.970840 14239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:36.971117 14239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:36.974402 14239 hybrid_clock.cc:648] HybridClock initialized: now 1790405976973844 us; error 564 us; skew 500 ppm
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:36.977689 14239 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:59:36.978227 14239 webserver.cc:460] Webserver started at http://127.1.74.66:37929/ using document root <none> and password file <none>
I20260926 06:59:36.978489 14239 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:36.978562 14239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:59:36 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:37.004437 14239 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 06:59:37.005120 14255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:37.005362 14239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:37.005471 14239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "c956c887663d4629b49a44cf56720ef1"
format_stamp: "Formatted at 2026-09-26 06:59:36 on dist-test-slave-sfvt"
server_key: "8c152674b896deb44468f271e1cb9d10"
server_key_iv: "9e5aa06cdfa7d803c0f3f8450dab04fe"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:37.005949 14239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:37.025740 14239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:37.027961 14239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:37.028128 14239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:37.028398 14239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:37.028797 14239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:37.028853 14239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:37.028893 14239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:37.028921 14239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:37.035853 14239 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:41495
I20260926 06:59:37.035924 14368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:41495 every 8 connection(s)
I20260926 06:59:37.036347 14239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:59:37 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:37.045569 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 14239
I20260926 06:59:37.045866 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.74.84:42615/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:46095
--local_ip_for_outbound_sockets=127.1.74.67
--tserver_master_addrs=127.1.74.126:33803
--webserver_port=35977
--webserver_interface=127.1.74.67
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.74.84:42615/kms with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:37.049145 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:37.037438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:59559 (local address 127.1.74.126:33803)
0926 06:59:37.037487 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:37.037489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:37.038033 (+ 544us) server_negotiation.cc:407] Connection header received
0926 06:59:37.038730 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:37.038732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:37.038750 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:37.038830 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:37.040021 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.040826 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.042395 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.042704 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.042812 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:37.045576 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:37.045578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:37.045578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:37.045587 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:37.047656 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:37.048159 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:37.048160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:37.048161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:37.048227 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:37.048478 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:37.048479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:37.048479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:37.048599 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:37.048676 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:37.048953 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 06:59:37.049042 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:59:37.049310 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:37.037426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33803 (local address 127.1.74.66:59559)
0926 06:59:37.037866 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:37.037914 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:37.038582 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:37.038911 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:37.038916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:37.039162 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:37.039915 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:37.039923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.040918 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.040919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:37.042196 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:37.042199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.042902 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.042904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:37.043062 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:37.043713 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:37.043724 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:37.045457 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:37.047744 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:37.047750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:37.047752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:37.048063 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:37.048301 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:37.048302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:37.048302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:37.048407 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:37.048667 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:37.048673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:37.048816 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 06:59:37.048919 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 06:59:37.049062 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":93,"threads_started":1}
I20260926 06:59:37.049753 14369 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33803
I20260926 06:59:37.049871 14369 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:37.050058 14369 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
W20260926 06:59:37.050678 14369 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:33803 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 06:59:37.050760 14369 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:37.050824 14369 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
I20260926 06:59:37.051095 14369 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:37.051164 14369 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
W20260926 06:59:37.051471 14369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:59:37.169514 14372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:37.169800 14372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:37.169842 14372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:37.171605 14372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:37.171664 14372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:37.171705 14372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:59:37.173933 14372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.74.84:42615/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:46095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=35977
--tserver_master_addrs=127.1.74.126:33803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:37.174211 14372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:37.174530 14372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:37.181300 14378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:37.181456 14380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:37.181723 14377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:37.182315 14372 server_base.cc:1034] running on GCE node
I20260926 06:59:37.182545 14372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:37.182823 14372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:37.183997 14372 hybrid_clock.cc:648] HybridClock initialized: now 1790405977183971 us; error 40 us; skew 500 ppm
Sep 26 06:59:37 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405977, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:37.187317 14372 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:59:37.187826 14372 webserver.cc:460] Webserver started at http://127.1.74.67:35977/ using document root <none> and password file <none>
I20260926 06:59:37.188102 14372 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:37.188174 14372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:59:37 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405977, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:37.216987 14372 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260926 06:59:37.218067 14389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:37.218261 14372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:59:37.218338 14372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "98dce2412a41471c8197c8a2eda22ed6"
format_stamp: "Formatted at 2026-09-26 06:59:36 on dist-test-slave-sfvt"
server_key: "7f8b5788d103551c62f91fcb59acaa1b"
server_key_iv: "4cdb79e046d7343974b0c1e92161e04c"
server_key_version: "kuduclusterkey@0"
I20260926 06:59:37.218824 14372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:37.240552 14372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:37.243031 14372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:37.243206 14372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:37.243474 14372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:37.243861 14372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:37.243913 14372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:37.243952 14372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:37.243978 14372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:37.251157 14372 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:46095
I20260926 06:59:37.251660 14372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:59:37.252667 14502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:46095 every 8 connection(s)
I20260926 06:59:37.253382 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 14372
Sep 26 06:59:37 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405977, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:37.265918 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:37.254350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.1.74.126:33803)
0926 06:59:37.256962 (+ 2612us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:37.256964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:37.256971 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:59:37.256987 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:37.256988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:37.256999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:37.257056 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:37.257855 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.258566 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.259674 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.259973 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.260047 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:37.261513 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:37.261514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:37.261514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:37.261521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:37.263583 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:37.264775 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:37.264776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:37.264776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:37.264852 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:37.265331 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:37.265332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:37.265332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:37.265437 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:37.265590 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:37.265703 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 06:59:37.265783 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":964,"thread_start_us":73,"threads_started":1}
I20260926 06:59:37.266396 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:37.253868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:36749 (local address 127.1.74.126:33803)
0926 06:59:37.253918 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:37.253920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:37.255176 (+ 1256us) server_negotiation.cc:407] Connection header received
0926 06:59:37.255922 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:37.255924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:37.255937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:37.256011 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:37.256922 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.257727 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.259178 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.259516 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.259558 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:37.262695 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:37.262697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:37.262697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:37.262705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:37.264258 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:37.264758 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:37.264760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:37.264760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:37.264835 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:37.265102 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:37.265103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:37.265104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:37.265245 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:37.265310 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:37.266236 (+ 926us) server_negotiation.cc:299] Negotiation successful
0926 06:59:37.266327 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:59:37.266419 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:37.253891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33803 (local address 127.1.74.67:36749)
0926 06:59:37.255010 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:37.255060 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:37.255812 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:37.256252 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:37.256257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:37.256498 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:37.256816 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:37.256822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.257842 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.257843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:37.259083 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:37.259086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:37.259671 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:37.259672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:37.259810 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:37.260618 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:37.260628 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:37.262587 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:37.264364 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:37.264371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:37.264373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:37.264667 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:37.264910 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:37.264911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:37.264911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:37.265025 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:37.265301 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:37.265307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:37.265473 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 06:59:37.266027 (+ 554us) client_negotiation.cc:240] Negotiation successful
0926 06:59:37.266181 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":86,"threads_started":1}
I20260926 06:59:37.267100 14503 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33803
I20260926 06:59:37.267201 14503 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:37.267427 14503 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
W20260926 06:59:37.268308 14503 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:33803 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 06:59:37.268383 14503 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:37.268452 14503 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
I20260926 06:59:37.268760 14503 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:37.268832 14503 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
W20260926 06:59:37.269086 14503 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 06:59:37 dist-test-slave-sfvt krb5kdc[13041](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790405977, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:37.853211 14235 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:37.853374 14235 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
I20260926 06:59:37.853762 14036 ts_manager.cc:194] Registered new tserver with Master: 7d25d06e07e046f2976f96e0105ecb74 (127.1.74.65:45461)
I20260926 06:59:37.854496 14036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:51145
I20260926 06:59:38.051653 14369 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:38.051810 14369 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
I20260926 06:59:38.052098 14036 ts_manager.cc:194] Registered new tserver with Master: c956c887663d4629b49a44cf56720ef1 (127.1.74.66:41495)
I20260926 06:59:38.052629 14036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:59559
Sep 26 06:59:38 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405977, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:38.269285 14503 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:38.269464 14503 heartbeater.cc:511] Master 127.1.74.126:33803 requested a full tablet report, sending...
I20260926 06:59:38.269791 14036 ts_manager.cc:194] Registered new tserver with Master: 98dce2412a41471c8197c8a2eda22ed6 (127.1.74.67:46095)
I20260926 06:59:38.270216 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:59:38.270292 14036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:36749
I20260926 06:59:38.279704 14525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.271410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.1.74.126:33803)
0926 06:59:38.271544 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:38.271546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:38.271582 (+ 36us) server_negotiation.cc:407] Connection header received
0926 06:59:38.271661 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:38.271662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:38.271675 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:38.271750 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:38.272445 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.273238 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.274333 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.274672 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.274744 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.276393 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:38.276395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:38.276395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:38.276403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:38.278297 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:38.278713 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:38.278714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:38.278715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:38.278877 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:38.279112 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:38.279113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:38.279113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:38.279281 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:38.279398 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:38.279509 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 06:59:38.279598 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260926 06:59:38.282200 14036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:59:38.284605 14036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:59:38.296335 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.287479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:46095 (local address 127.0.0.1:36020)
0926 06:59:38.287791 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:38.287828 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:38.287891 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:38.291849 (+ 3958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:38.291852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:38.291855 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:38.291946 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.291948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.293545 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.293546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.294703 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.294706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.296038 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.296039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.296125 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.296143 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 06:59:38.296194 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:59:38.296245 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":6,"thread_start_us":101,"threads_started":1}
I20260926 06:59:38.296770 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.288079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.1.74.67:46095)
0926 06:59:38.291670 (+ 3591us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:38.291676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:38.291686 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:59:38.291720 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:38.291722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:38.291738 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:38.292024 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:38.292099 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.293447 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.294808 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.295935 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.296552 (+ 617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.296620 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:38.296647 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 06:59:38.296688 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3558,"thread_start_us":66,"threads_started":1}
I20260926 06:59:38.297439 14437 tablet_service.cc:1467] Processing CreateTablet for tablet a3094919895e4c16800911e57e087755 (DEFAULT_TABLE table=test-table [id=8ba21e83612c4aa189bdad22d4d6f16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:59:38.297775 14437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3094919895e4c16800911e57e087755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:38.299003 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.288053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.1.74.65:45461)
0926 06:59:38.288190 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:38.288195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:38.291598 (+ 3403us) server_negotiation.cc:407] Connection header received
0926 06:59:38.291634 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:38.291636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:38.291651 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:38.291721 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:38.292025 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.293195 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.297817 (+ 4622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.298563 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.298611 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.298662 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:38.298883 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 06:59:38.298927 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":53,"threads_started":1}
I20260926 06:59:38.299218 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.287725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:45461 (local address 127.0.0.1:50274)
0926 06:59:38.291519 (+ 3794us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:38.291527 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:38.291570 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:38.291849 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:38.291852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:38.291858 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:38.291945 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.291947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.293290 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.293291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.297511 (+ 4220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.297513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.298721 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.298722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.298795 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.298799 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:59:38.299109 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 06:59:38.299150 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3780,"mutex_wait_us":4,"thread_start_us":58,"threads_started":1}
I20260926 06:59:38.300400 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.287482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:41495 (local address 127.0.0.1:53336)
0926 06:59:38.291363 (+ 3881us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:38.291374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:38.291444 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:38.294224 (+ 2780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:38.294225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:38.294227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:38.294265 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.294266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.295540 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.295541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.297721 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.297722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.300175 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.300176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.300239 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.300243 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:59:38.300288 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 06:59:38.300323 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3862,"thread_start_us":115,"threads_started":1}
I20260926 06:59:38.300638 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.288239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.1.74.66:41495)
0926 06:59:38.288374 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:38.288378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:38.291457 (+ 3079us) server_negotiation.cc:407] Connection header received
0926 06:59:38.291502 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:38.291505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:38.291522 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:38.294146 (+ 2624us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:38.294341 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.295450 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.299299 (+ 3849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.300027 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.300094 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.300141 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:38.300519 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 06:59:38.300565 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260926 06:59:38.301159 14169 tablet_service.cc:1467] Processing CreateTablet for tablet a3094919895e4c16800911e57e087755 (DEFAULT_TABLE table=test-table [id=8ba21e83612c4aa189bdad22d4d6f16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:59:38.301278 14287 tablet_service.cc:1467] Processing CreateTablet for tablet a3094919895e4c16800911e57e087755 (DEFAULT_TABLE table=test-table [id=8ba21e83612c4aa189bdad22d4d6f16a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:59:38.301465 14169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3094919895e4c16800911e57e087755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:38.301606 14287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3094919895e4c16800911e57e087755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:38.304816 14537 tablet_bootstrap.cc:492] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74: Bootstrap starting.
I20260926 06:59:38.305408 14537 tablet_bootstrap.cc:654] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:38.305713 14537 log.cc:824] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:38.306591 14537 tablet_bootstrap.cc:492] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74: No bootstrap required, opened a new log
I20260926 06:59:38.306654 14537 ts_tablet_manager.cc:1397] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:59:38.307437 14536 tablet_bootstrap.cc:492] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: Bootstrap starting.
I20260926 06:59:38.308115 14536 tablet_bootstrap.cc:654] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:38.308552 14536 log.cc:824] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:38.309175 14537 raft_consensus.cc:348] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } }
I20260926 06:59:38.309340 14537 raft_consensus.cc:374] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:38.309378 14537 raft_consensus.cc:729] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d25d06e07e046f2976f96e0105ecb74, State: Initialized, Role: FOLLOWER
I20260926 06:59:38.309556 14537 consensus_queue.cc:260] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } }
I20260926 06:59:38.309854 14235 heartbeater.cc:503] Master 127.1.74.126:33803 was elected leader, sending a full tablet report...
I20260926 06:59:38.309989 14537 ts_tablet_manager.cc:1428] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:59:38.311419 14538 tablet_bootstrap.cc:492] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: Bootstrap starting.
I20260926 06:59:38.311950 14538 tablet_bootstrap.cc:654] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:38.312224 14538 log.cc:824] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:38.313023 14538 tablet_bootstrap.cc:492] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: No bootstrap required, opened a new log
I20260926 06:59:38.313088 14538 ts_tablet_manager.cc:1397] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:59:38.315593 14538 raft_consensus.cc:348] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } }
I20260926 06:59:38.315747 14538 raft_consensus.cc:374] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:38.315786 14538 raft_consensus.cc:729] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c956c887663d4629b49a44cf56720ef1, State: Initialized, Role: FOLLOWER
I20260926 06:59:38.315881 14536 tablet_bootstrap.cc:492] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: No bootstrap required, opened a new log
I20260926 06:59:38.315939 14536 ts_tablet_manager.cc:1397] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260926 06:59:38.315909 14538 consensus_queue.cc:260] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } }
I20260926 06:59:38.316699 14538 ts_tablet_manager.cc:1428] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 06:59:38.316839 14369 heartbeater.cc:503] Master 127.1.74.126:33803 was elected leader, sending a full tablet report...
I20260926 06:59:38.318461 14536 raft_consensus.cc:348] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } }
I20260926 06:59:38.318606 14536 raft_consensus.cc:374] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:38.318637 14536 raft_consensus.cc:729] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98dce2412a41471c8197c8a2eda22ed6, State: Initialized, Role: FOLLOWER
I20260926 06:59:38.318745 14536 consensus_queue.cc:260] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } }
I20260926 06:59:38.318993 14536 ts_tablet_manager.cc:1428] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:59:38.319041 14503 heartbeater.cc:503] Master 127.1.74.126:33803 was elected leader, sending a full tablet report...
W20260926 06:59:38.337476 14236 tablet.cc:2369] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:59:38.381263 14544 raft_consensus.cc:482] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:59:38.381413 14544 raft_consensus.cc:504] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } }
I20260926 06:59:38.382494 14544 leader_election.cc:288] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c956c887663d4629b49a44cf56720ef1 (127.1.74.66:41495), 7d25d06e07e046f2976f96e0105ecb74 (127.1.74.65:45461)
I20260926 06:59:38.386813 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.382190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:53893 (local address 127.1.74.66:41495)
0926 06:59:38.382235 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:38.382237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:38.382680 (+ 443us) server_negotiation.cc:407] Connection header received
0926 06:59:38.382813 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:38.382814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:38.382827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:38.382864 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:38.383595 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.384357 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.385867 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.386416 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.386465 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.386475 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:38.386689 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 06:59:38.386728 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:59:38.387064 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.381967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:41495 (local address 127.1.74.67:53893)
0926 06:59:38.382614 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:38.382625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:38.382719 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:38.382950 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:38.382951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:38.382954 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:38.383037 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.383039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.384466 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.384467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.385752 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.385755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.386544 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.386545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.386612 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.386616 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:59:38.386930 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 06:59:38.386972 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":98,"threads_started":1}
I20260926 06:59:38.387624 14312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3094919895e4c16800911e57e087755" candidate_uuid: "98dce2412a41471c8197c8a2eda22ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c956c887663d4629b49a44cf56720ef1" is_pre_election: true
I20260926 06:59:38.387811 14312 raft_consensus.cc:2454] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98dce2412a41471c8197c8a2eda22ed6 in term 0.
I20260926 06:59:38.388073 14392 leader_election.cc:302] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98dce2412a41471c8197c8a2eda22ed6, c956c887663d4629b49a44cf56720ef1; no voters:
I20260926 06:59:38.388208 14544 raft_consensus.cc:2781] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:59:38.388239 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.382510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:34351 (local address 127.1.74.65:45461)
0926 06:59:38.382557 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:38.382558 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:38.384147 (+ 1589us) server_negotiation.cc:407] Connection header received
0926 06:59:38.384261 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:38.384262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:38.384276 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:38.384306 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:38.384551 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.385216 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.386545 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.387097 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.387146 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.387156 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:38.388138 (+ 982us) server_negotiation.cc:299] Negotiation successful
0926 06:59:38.388180 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:59:38.388252 14544 raft_consensus.cc:482] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:59:38.388275 14544 raft_consensus.cc:3037] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:38.388590 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.382391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:45461 (local address 127.1.74.67:34351)
0926 06:59:38.384063 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:38.384075 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:38.384178 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:38.384372 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:38.384373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:38.384374 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:38.384461 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.384463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.385322 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.385323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.386436 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:38.386439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.387985 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.387986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:38.388055 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.388059 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:59:38.388479 (+ 420us) client_negotiation.cc:240] Negotiation successful
0926 06:59:38.388520 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1652,"thread_start_us":75,"threads_started":1}
I20260926 06:59:38.389183 14544 raft_consensus.cc:504] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } }
I20260926 06:59:38.389320 14544 leader_election.cc:288] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [CANDIDATE]: Term 1 election: Requested vote from peers c956c887663d4629b49a44cf56720ef1 (127.1.74.66:41495), 7d25d06e07e046f2976f96e0105ecb74 (127.1.74.65:45461)
I20260926 06:59:38.389580 14189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3094919895e4c16800911e57e087755" candidate_uuid: "98dce2412a41471c8197c8a2eda22ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d25d06e07e046f2976f96e0105ecb74" is_pre_election: true
I20260926 06:59:38.389721 14189 raft_consensus.cc:2454] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98dce2412a41471c8197c8a2eda22ed6 in term 0.
I20260926 06:59:38.390151 14189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3094919895e4c16800911e57e087755" candidate_uuid: "98dce2412a41471c8197c8a2eda22ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d25d06e07e046f2976f96e0105ecb74"
I20260926 06:59:38.390228 14189 raft_consensus.cc:3037] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:38.391542 14312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3094919895e4c16800911e57e087755" candidate_uuid: "98dce2412a41471c8197c8a2eda22ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c956c887663d4629b49a44cf56720ef1"
I20260926 06:59:38.391584 14189 raft_consensus.cc:2454] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98dce2412a41471c8197c8a2eda22ed6 in term 1.
I20260926 06:59:38.391628 14312 raft_consensus.cc:3037] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:38.391748 14392 leader_election.cc:302] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d25d06e07e046f2976f96e0105ecb74, 98dce2412a41471c8197c8a2eda22ed6; no voters:
I20260926 06:59:38.391822 14544 raft_consensus.cc:2781] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:59:38.391952 14544 raft_consensus.cc:686] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 1 LEADER]: Becoming Leader. State: Replica: 98dce2412a41471c8197c8a2eda22ed6, State: Running, Role: LEADER
I20260926 06:59:38.392032 14544 consensus_queue.cc:237] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } }
I20260926 06:59:38.392400 14312 raft_consensus.cc:2454] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98dce2412a41471c8197c8a2eda22ed6 in term 1.
I20260926 06:59:38.392877 14036 catalog_manager.cc:5495] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98dce2412a41471c8197c8a2eda22ed6 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "98dce2412a41471c8197c8a2eda22ed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98dce2412a41471c8197c8a2eda22ed6" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 } health_report { overall_health: UNKNOWN } } }
I20260926 06:59:38.412268 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:38.408663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.1.74.67:46095)
0926 06:59:38.408742 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:38.408745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:38.408754 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:59:38.408805 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:38.408806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:38.408829 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:38.408918 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:59:38.409099 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.409934 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.411113 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:38.411466 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:38.411529 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:38.411713 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:59:38.411845 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:59:38.411938 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:38.412167 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 06:59:38.412190 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 06:59:38.415299 14189 raft_consensus.cc:1261] T a3094919895e4c16800911e57e087755 P 7d25d06e07e046f2976f96e0105ecb74 [term 1 FOLLOWER]: Refusing update from remote peer 98dce2412a41471c8197c8a2eda22ed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:59:38.415642 14547 consensus_queue.cc:1035] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d25d06e07e046f2976f96e0105ecb74" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 45461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:59:38.417053 14312 raft_consensus.cc:1261] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [term 1 FOLLOWER]: Refusing update from remote peer 98dce2412a41471c8197c8a2eda22ed6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:59:38.417510 14547 consensus_queue.cc:1035] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c956c887663d4629b49a44cf56720ef1" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:59:38.420312 14551 mvcc.cc:205] Tried to move back new op lower bound from 7333502887585947648 to 7333502887494242304. Current Snapshot: MvccSnapshot[applied={T|T < 7333502887585947648}]
I20260926 06:59:38.420786 14554 mvcc.cc:205] Tried to move back new op lower bound from 7333502887585947648 to 7333502887494242304. Current Snapshot: MvccSnapshot[applied={T|T < 7333502887585947648}]
I20260926 06:59:38.421905 14552 mvcc.cc:205] Tried to move back new op lower bound from 7333502887585947648 to 7333502887494242304. Current Snapshot: MvccSnapshot[applied={T|T < 7333502887585947648}]
I20260926 06:59:38.424268 14036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:59:38.424365 14036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:59:38.425102 14036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84489a825a2e43aa9b1b037e7e4223f5: Sending DeleteTablet for 3 replicas of tablet a3094919895e4c16800911e57e087755
I20260926 06:59:38.425508 14287 tablet_service.cc:1514] Processing DeleteTablet for tablet a3094919895e4c16800911e57e087755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:38 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:53336
I20260926 06:59:38.425716 14561 tablet_replica.cc:330] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: stopping tablet replica
I20260926 06:59:38.425716 14437 tablet_service.cc:1514] Processing DeleteTablet for tablet a3094919895e4c16800911e57e087755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:38 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:36020
I20260926 06:59:38.425814 14561 raft_consensus.cc:2229] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:59:38.425808 14169 tablet_service.cc:1514] Processing DeleteTablet for tablet a3094919895e4c16800911e57e087755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:38 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:50274
I20260926 06:59:38.425922 14561 raft_consensus.cc:2258] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 26 06:59:38 dist-test-slave-sfvt krb5kdc[13041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405977, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for HTTP/127.1.74.84@KRBTEST.COM
I20260926 06:59:38.426528 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 14075
I20260926 06:59:38.430876 14561 ts_tablet_manager.cc:1905] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:59:38.433638 14561 ts_tablet_manager.cc:1918] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:59:38.433748 14561 log.cc:1196] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/a3094919895e4c16800911e57e087755
I20260926 06:59:38.434026 14561 ts_tablet_manager.cc:1939] T a3094919895e4c16800911e57e087755 P c956c887663d4629b49a44cf56720ef1: Deleting consensus metadata
I20260926 06:59:38.434445 14023 catalog_manager.cc:4841] TS c956c887663d4629b49a44cf56720ef1 (127.1.74.66:41495): tablet a3094919895e4c16800911e57e087755 (table test-table [id=8ba21e83612c4aa189bdad22d4d6f16a]) successfully deleted
I20260926 06:59:38.436797 14562 tablet_replica.cc:330] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: stopping tablet replica
I20260926 06:59:38.436915 14562 raft_consensus.cc:2229] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:59:38.437047 14562 raft_consensus.cc:2258] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 06:59:38.437877 14023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:45461 (error 108)
I20260926 06:59:38.438438 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 14239
I20260926 06:59:38.442056 14562 ts_tablet_manager.cc:1905] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:59:38.446153 14562 ts_tablet_manager.cc:1918] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:59:38.446238 14562 log.cc:1196] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/a3094919895e4c16800911e57e087755
I20260926 06:59:38.446537 14562 ts_tablet_manager.cc:1939] T a3094919895e4c16800911e57e087755 P 98dce2412a41471c8197c8a2eda22ed6: Deleting consensus metadata
I20260926 06:59:38.446955 14024 catalog_manager.cc:4841] TS 98dce2412a41471c8197c8a2eda22ed6 (127.1.74.67:46095): tablet a3094919895e4c16800911e57e087755 (table test-table [id=8ba21e83612c4aa189bdad22d4d6f16a]) successfully deleted
I20260926 06:59:38.447149 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 14372
I20260926 06:59:38.451339 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 14004
I20260926 06:59:38.458485 1321 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 06:59:38.803464 1321 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 06:59:38.803563 1321 mini_ranger.cc:66] Stopping Ranger...
I20260926 06:59:39.147871 1321 mini_ranger.cc:68] Stopped Ranger
2026-09-26 06:59:39.148 UTC [13071] LOG: received smart shutdown request
2026-09-26 06:59:39.151 UTC [13071] LOG: background worker "logical replication launcher" (PID 13080) exited with exit code 1
2026-09-26 06:59:39.152 UTC [13075] LOG: shutting down
2026-09-26 06:59:39.322 UTC [13071] LOG: database system is shut down
2026-09-26T06:59:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47286 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:59:39 dist-test-slave-sfvt krb5kdc[14574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:59:39 dist-test-slave-sfvt krb5kdc[14574](info): set up 2 sockets
Sep 26 06:59:39 dist-test-slave-sfvt krb5kdc[14574](info): commencing operation
krb5kdc: starting...
W20260926 06:59:40.479530 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:59:40 dist-test-slave-sfvt krb5kdc[14574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:59:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:59:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:59:40.610600 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45759
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:45759,127.1.74.125:35661,127.1.74.124:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:40.690644 14587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:40.690925 14587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:40.690965 14587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:40.692134 14587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:59:40.692240 14587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:40.692267 14587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:40.692296 14587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:59:40.692327 14587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:59:40.693852 14587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:45759,127.1.74.125:35661,127.1.74.124:43781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:40.694156 14587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:40.694443 14587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:40.699182 14592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:40.699173 14593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:40.699867 14587 server_base.cc:1034] running on GCE node
W20260926 06:59:40.699208 14595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:40.700176 14587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:40.700417 14587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:40.701555 14587 hybrid_clock.cc:648] HybridClock initialized: now 1790405980701538 us; error 39 us; skew 500 ppm
Sep 26 06:59:40 dist-test-slave-sfvt krb5kdc[14574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:40.707532 14587 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:59:40.707939 14587 webserver.cc:460] Webserver started at http://127.1.74.126:35417/ using document root <none> and password file <none>
I20260926 06:59:40.708118 14587 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:40.708158 14587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:40.708247 14587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:59:40.709299 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e"
format_stamp: "Formatted at 2026-09-26 06:59:40 on dist-test-slave-sfvt"
server_key: "929f5935179415da590502e8db93b6d0"
server_key_iv: "eb308da5d79a774eb186918c8017bdd2"
server_key_version: "encryptionkey@0"
I20260926 06:59:40.709599 14587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e"
format_stamp: "Formatted at 2026-09-26 06:59:40 on dist-test-slave-sfvt"
server_key: "929f5935179415da590502e8db93b6d0"
server_key_iv: "eb308da5d79a774eb186918c8017bdd2"
server_key_version: "encryptionkey@0"
I20260926 06:59:40.710839 14587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:59:40.711558 14602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:40.711728 14587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:40.711814 14587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e"
format_stamp: "Formatted at 2026-09-26 06:59:40 on dist-test-slave-sfvt"
server_key: "929f5935179415da590502e8db93b6d0"
server_key_iv: "eb308da5d79a774eb186918c8017bdd2"
server_key_version: "encryptionkey@0"
I20260926 06:59:40.711869 14587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:40.722962 14587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:40.725039 14587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:40.725178 14587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:40.728917 14587 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:45759
I20260926 06:59:40.729007 14654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:45759 every 8 connection(s)
I20260926 06:59:40.729314 14587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:59:40.730180 14655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:40.731832 14655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:59:40.735379 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 14587
I20260926 06:59:40.735469 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:59:40.735658 1321 external_mini_cluster.cc:1428] Setting key b8b5731f3dbe3ff0732f28c2f1b99cfa
Sep 26 06:59:40 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:40.741942 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.732502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45759 (local address 127.0.0.1:46734)
0926 06:59:40.732667 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.732714 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:40.733141 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:40.733396 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:40.733399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:40.733641 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:40.733793 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:40.733797 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.734762 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.734763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:40.735593 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:40.735596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.736145 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.736146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:40.736212 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:40.736811 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:40.736822 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:40.738251 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:40.740458 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:40.740464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:40.740466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:40.740660 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:40.740933 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:40.740933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:40.740933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:40.741009 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:40.741459 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:40.741463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:40.741526 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 06:59:40.741599 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 06:59:40.741727 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260926 06:59:40.741952 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.732538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.1.74.126:45759)
0926 06:59:40.732756 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:40.732760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:40.732787 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:59:40.733272 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:40.733273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:40.733290 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:40.733368 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:40.733872 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.734681 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.735671 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.736043 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.736175 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:40.738342 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:40.738349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:40.738350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:40.738357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:40.740371 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:40.740746 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:40.740749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:40.740750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:40.740858 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:40.741075 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:40.741077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:40.741077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:40.741389 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:40.741482 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:40.741656 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 06:59:40.741817 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":192,"thread_start_us":119,"threads_started":1}
I20260926 06:59:40.742808 14655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:59:40.743185 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.743050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:35661 (local address 127.0.0.1:44346)
0926 06:59:40.743103 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.743153 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 06:59:40.743366 14604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
I20260926 06:59:40.743798 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.743698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:35661 (local address 127.0.0.1:44360)
0926 06:59:40.743747 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.743776 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 06:59:40.743930 14655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.125:35661: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.74.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab.
Entry for principal kudu/127.1.74.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab.
I20260926 06:59:40.762596 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.125:35661
--webserver_interface=127.1.74.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab
--principal=kudu/127.1.74.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:45759,127.1.74.125:35661,127.1.74.124:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:40.793177 14655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } attempt: 1
I20260926 06:59:40.793700 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.793531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:35661 (local address 127.0.0.1:44370)
0926 06:59:40.793611 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.793661 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 06:59:40.794083 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.793990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:35661 (local address 127.0.0.1:44378)
0926 06:59:40.794034 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.794063 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 06:59:40.794209 14655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.125:35661: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
W20260926 06:59:40.843178 14662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:40.843484 14662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:40.843519 14662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:40.845173 14662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:59:40.845233 14662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:40.845255 14662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:40.845273 14662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:59:40.845292 14662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:59:40.847015 14662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:45759,127.1.74.125:35661,127.1.74.124:43781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab
--principal=kudu/127.1.74.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.125:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.74.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:40.847293 14662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:40.847496 14662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:40.852250 14668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:40.852272 14667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:40.852370 14662 server_base.cc:1034] running on GCE node
W20260926 06:59:40.852250 14670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:40.852728 14662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:40.852960 14662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:40.854094 14662 hybrid_clock.cc:648] HybridClock initialized: now 1790405980854076 us; error 33 us; skew 500 ppm
Sep 26 06:59:40 dist-test-slave-sfvt krb5kdc[14574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:40.856532 14662 init.cc:377] Logged in from keytab as kudu/127.1.74.125@KRBTEST.COM (short username kudu)
I20260926 06:59:40.856966 14662 webserver.cc:460] Webserver started at http://127.1.74.125:35203/ using document root <none> and password file <none>
I20260926 06:59:40.857151 14662 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:40.857191 14662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:40.857293 14662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:59:40.858287 14662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/data/instance:
uuid: "86d88156fd684684ba84da946582f495"
format_stamp: "Formatted at 2026-09-26 06:59:40 on dist-test-slave-sfvt"
server_key: "5233cacf13fce241b7943bccc9f21476"
server_key_iv: "45bce637c587aab594fc952b3da507af"
server_key_version: "encryptionkey@0"
I20260926 06:59:40.858580 14662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/wal/instance:
uuid: "86d88156fd684684ba84da946582f495"
format_stamp: "Formatted at 2026-09-26 06:59:40 on dist-test-slave-sfvt"
server_key: "5233cacf13fce241b7943bccc9f21476"
server_key_iv: "45bce637c587aab594fc952b3da507af"
server_key_version: "encryptionkey@0"
I20260926 06:59:40.859907 14662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 06:59:40.860585 14677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:40.860764 14662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:59:40.860833 14662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/wal
uuid: "86d88156fd684684ba84da946582f495"
format_stamp: "Formatted at 2026-09-26 06:59:40 on dist-test-slave-sfvt"
server_key: "5233cacf13fce241b7943bccc9f21476"
server_key_iv: "45bce637c587aab594fc952b3da507af"
server_key_version: "encryptionkey@0"
I20260926 06:59:40.860891 14662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:40.862371 14655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } attempt: 2
I20260926 06:59:40.862768 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.862620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:35661 (local address 127.0.0.1:44388)
0926 06:59:40.862681 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.862730 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 06:59:40.863112 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.863005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:35661 (local address 127.0.0.1:44394)
0926 06:59:40.863056 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.863088 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 06:59:40.863206 14655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.125:35661: Network error: Client connection negotiation failed: client connection to 127.1.74.125:35661: connect: Connection refused (error 111)
I20260926 06:59:40.872648 14662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:40.874459 14662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:40.874614 14662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:40.878813 14662 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.125:35661
I20260926 06:59:40.878856 14729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.125:35661 every 8 connection(s)
I20260926 06:59:40.879230 14662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
I20260926 06:59:40.880061 14730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:40.881553 14730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:59:40 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:40.887305 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 14662
I20260926 06:59:40.887425 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-1/wal/instance
I20260926 06:59:40.887600 1321 external_mini_cluster.cc:1428] Setting key 7819e0e539d6c86b9dbe11e6e3d83e5c
I20260926 06:59:40.890311 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.882225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.1.74.126:45759)
0926 06:59:40.882286 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:40.882289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:40.882475 (+ 186us) server_negotiation.cc:407] Connection header received
0926 06:59:40.882932 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:40.882933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:40.882941 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:40.882999 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:40.883645 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.884122 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.885068 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.885332 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.885399 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:40.887574 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:40.887575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:40.887576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:40.887590 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:40.888948 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:40.889354 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:40.889355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:40.889356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:40.889415 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:40.889674 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:40.889674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:40.889675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:40.889789 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:40.889865 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:40.890173 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 06:59:40.890243 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:59:40.890597 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.882196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45759 (local address 127.0.0.1:46736)
0926 06:59:40.882352 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.882392 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:40.882849 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:40.883051 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:40.883055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:40.883380 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:40.883572 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:40.883577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.884198 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.884198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:40.884970 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:40.884972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.885412 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.885413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:40.885509 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:40.886029 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:40.886037 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:40.887490 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:40.889037 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:40.889045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:40.889048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:40.889276 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:40.889482 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:40.889483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:40.889484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:40.889600 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:40.889857 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:40.889864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:40.890026 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 06:59:40.890138 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 06:59:40.890317 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260926 06:59:40.891222 14730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.74.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.124@KRBTEST.COM" created.
Sep 26 06:59:40 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 06:59:40.907286 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.891556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:35661 (local address 127.0.0.1:44404)
0926 06:59:40.891612 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.891625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:40.891670 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:40.891888 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:40.891889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:40.892052 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:40.892111 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:40.892113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.892959 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.892960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:40.893801 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:40.893804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.894241 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.894242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:40.894321 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:40.894757 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:40.894762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:40.903763 (+ 9001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:40.905980 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:40.905981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:40.905982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:40.906236 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:40.906487 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:40.906488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:40.906488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:40.906539 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:40.906902 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:40.906903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:40.906956 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 06:59:40.907033 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:59:40.907149 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 06:59:40.907613 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.891591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.1.74.125:35661)
0926 06:59:40.891737 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:40.891741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:40.891759 (+ 18us) server_negotiation.cc:407] Connection header received
0926 06:59:40.891785 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:40.891786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:40.891803 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:40.891864 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:40.892187 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.892875 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.893887 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.894163 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.894213 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:40.903894 (+ 9681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:40.903908 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:40.903909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:40.903916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:40.905897 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:40.906355 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:40.906359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:40.906359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:40.906420 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:40.906599 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:40.906600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:40.906600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:40.906801 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:40.906885 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:40.907089 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 06:59:40.907301 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260926 06:59:40.908267 14730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:59:40.909200 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.908512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51590)
0926 06:59:40.909125 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.909175 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593}
W20260926 06:59:40.909353 14681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
I20260926 06:59:40.909855 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.909764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51604)
0926 06:59:40.909801 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.909832 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 06:59:40.909987 14730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:43781: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab.
Entry for principal kudu/127.1.74.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab.
I20260926 06:59:40.918277 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.124:43781
--webserver_interface=127.1.74.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab
--principal=kudu/127.1.74.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:45759,127.1.74.125:35661,127.1.74.124:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:40.954344 14655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } attempt: 3
Sep 26 06:59:40 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 06:59:40.959179 14730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } attempt: 1
I20260926 06:59:40.959607 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.959492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51618)
0926 06:59:40.959545 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.959580 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 06:59:40.959945 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.959852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51624)
0926 06:59:40.959897 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.959926 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 06:59:40.960053 14730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:43781: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
I20260926 06:59:40.961967 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.954731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:35661 (local address 127.0.0.1:44420)
0926 06:59:40.954793 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.954808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:40.954877 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:40.955073 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:40.955074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:40.955220 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:40.955343 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:40.955345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.956296 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.956296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:40.956809 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:40.956810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.957325 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.957326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:40.957388 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:40.957862 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:40.957867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:40.959018 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:40.960977 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:40.960979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:40.960979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:40.961143 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:40.961366 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:40.961367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:40.961367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:40.961415 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:40.961676 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:40.961676 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:40.961713 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 06:59:40.961808 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 06:59:40.961904 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 06:59:40.961987 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.954858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.1.74.125:35661)
0926 06:59:40.954929 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:40.954931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:40.954942 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:59:40.954972 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:40.954973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:40.954984 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:40.955061 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:40.955429 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.956199 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.956906 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:40.957244 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:40.957315 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:40.959103 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:40.959104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:40.959105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:40.959112 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:40.960893 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:40.961226 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:40.961227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:40.961228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:40.961296 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:40.961484 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:40.961485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:40.961486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:40.961616 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:40.961683 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:40.961831 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 06:59:40.961905 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:59:40.962561 14655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:59:40.962910 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.962795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51640)
0926 06:59:40.962857 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.962885 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 06:59:40.963217 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.963156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51646)
0926 06:59:40.963180 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.963204 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 06:59:40.963354 14655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:43781: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
I20260926 06:59:40.994529 14655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } attempt: 1
I20260926 06:59:40.995072 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.994930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51660)
0926 06:59:40.995003 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.995046 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 06:59:40.995520 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:40.995426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51670)
0926 06:59:40.995478 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:40.995503 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 06:59:40.995631 14655 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:43781: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
W20260926 06:59:41.001199 14737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:41.001384 14737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:41.001406 14737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:41.002485 14737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:59:41.002518 14737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:41.002532 14737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:41.002545 14737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:59:41.002557 14737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:59:41.004014 14737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:45759,127.1.74.125:35661,127.1.74.124:43781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab
--principal=kudu/127.1.74.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.124:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.74.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:41.004199 14737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:41.004359 14737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:41.009198 14743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:41.009295 14737 server_base.cc:1034] running on GCE node
W20260926 06:59:41.009301 14742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:41.009198 14745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:41.009660 14737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:41.009904 14737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:41.011041 14737 hybrid_clock.cc:648] HybridClock initialized: now 1790405981011029 us; error 27 us; skew 500 ppm
Sep 26 06:59:41 dist-test-slave-sfvt krb5kdc[14574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:41.013430 14737 init.cc:377] Logged in from keytab as kudu/127.1.74.124@KRBTEST.COM (short username kudu)
I20260926 06:59:41.013903 14737 webserver.cc:460] Webserver started at http://127.1.74.124:46481/ using document root <none> and password file <none>
I20260926 06:59:41.014122 14737 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:41.014185 14737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:41.014351 14737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:59:41.015486 14737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/data/instance:
uuid: "d1d2205286a842b0a9578edaeffa6b05"
format_stamp: "Formatted at 2026-09-26 06:59:41 on dist-test-slave-sfvt"
server_key: "31ec9f8b3577360bd100821e96d60e63"
server_key_iv: "f6d681b2a19970fc6c990addf1be197d"
server_key_version: "encryptionkey@0"
I20260926 06:59:41.015831 14737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/wal/instance:
uuid: "d1d2205286a842b0a9578edaeffa6b05"
format_stamp: "Formatted at 2026-09-26 06:59:41 on dist-test-slave-sfvt"
server_key: "31ec9f8b3577360bd100821e96d60e63"
server_key_iv: "f6d681b2a19970fc6c990addf1be197d"
server_key_version: "encryptionkey@0"
I20260926 06:59:41.017087 14737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:41.017773 14752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:41.017938 14737 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:59:41.018033 14737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/wal
uuid: "d1d2205286a842b0a9578edaeffa6b05"
format_stamp: "Formatted at 2026-09-26 06:59:41 on dist-test-slave-sfvt"
server_key: "31ec9f8b3577360bd100821e96d60e63"
server_key_iv: "f6d681b2a19970fc6c990addf1be197d"
server_key_version: "encryptionkey@0"
I20260926 06:59:41.018111 14737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:41.028191 14730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } attempt: 2
I20260926 06:59:41.028613 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.028487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51682)
0926 06:59:41.028549 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:41.028585 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 06:59:41.028944 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.028855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51692)
0926 06:59:41.028900 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:41.028924 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:59:41.029022 14730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:43781: Network error: Client connection negotiation failed: client connection to 127.1.74.124:43781: connect: Connection refused (error 111)
I20260926 06:59:41.033244 14737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:41.035051 14737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:41.035210 14737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:41.039021 14737 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.124:43781
I20260926 06:59:41.039067 14804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.124:43781 every 8 connection(s)
I20260926 06:59:41.039479 14737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
I20260926 06:59:41.040328 14805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:41.041884 14805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:59:41.043033 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 14737
I20260926 06:59:41.043115 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790405833506994-1321-0/minicluster-data/master-2/wal/instance
I20260926 06:59:41.043358 1321 external_mini_cluster.cc:1428] Setting key 1bc6b5a11f5d1c21fb2aa834bcfc2449
Sep 26 06:59:41 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
Sep 26 06:59:41 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
I20260926 06:59:41.052983 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.042603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.1.74.126:45759)
0926 06:59:41.042679 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.042682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.042960 (+ 278us) server_negotiation.cc:407] Connection header received
0926 06:59:41.043705 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.043706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.043730 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.043803 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.044602 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.045270 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.046518 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.046761 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.046832 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.049235 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.049236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.049236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.049244 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.051561 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.052010 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.052011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.052011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.052073 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.052382 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.052383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.052383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.052519 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.052588 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.052792 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.052876 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:59:41.053198 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.042588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45759 (local address 127.0.0.1:46746)
0926 06:59:41.042800 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:41.042837 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:41.043589 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:41.043879 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:41.043883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:41.044272 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:41.044505 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.044512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.045350 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.045351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.046438 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.046441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.046851 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.046851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.046967 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.047492 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:41.047501 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:41.049127 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:41.051639 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.051644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.051646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.051942 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.052136 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.052137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.052137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.052291 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.052585 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:41.052594 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:41.052647 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 06:59:41.052762 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 06:59:41.052938 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":122,"thread_start_us":101,"threads_started":1}
I20260926 06:59:41.053207 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.043741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.1.74.124:43781)
0926 06:59:41.043892 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.043896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.043968 (+ 72us) server_negotiation.cc:407] Connection header received
0926 06:59:41.044027 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.044028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.044046 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.044109 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.044864 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.045817 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.046774 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.047108 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.047194 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.049752 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.049758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.049758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.049780 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.051646 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.052018 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.052021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.052022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.052111 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.052382 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.052383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.052383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.052652 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.052810 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.052882 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.053109 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260926 06:59:41.053706 14805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:59:41 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
Sep 26 06:59:41 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:41.061026 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.054080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.1.74.125:35661)
0926 06:59:41.054125 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.054127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.054136 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:59:41.054182 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.054183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.054191 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.054253 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.054540 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.055043 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.055766 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.055970 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.056042 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.057830 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.057831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.057831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.057837 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.059864 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.060233 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.060234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.060234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.060282 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.060496 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.060497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.060498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.060638 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.060716 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.060872 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.060939 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:59:41.061112 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.053964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:35661 (local address 127.0.0.1:44436)
0926 06:59:41.054000 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:41.054020 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:41.054056 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:41.054270 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:41.054271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:41.054395 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:41.054438 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.054439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.055133 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.055134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.055674 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.055675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.056051 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.056051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.056121 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.056505 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:41.056510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:41.057749 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:41.059950 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.059951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.059951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.060141 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.060349 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.060350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.060350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.060415 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.060718 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:41.060719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:41.060760 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 06:59:41.060847 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 06:59:41.060934 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 06:59:41.061666 14805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:59:41.061702 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.054085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.1.74.126:45759)
0926 06:59:41.054121 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.054122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.054231 (+ 109us) server_negotiation.cc:407] Connection header received
0926 06:59:41.054299 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.054300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.054309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.054352 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.054887 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.055393 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.056043 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.056277 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.056340 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.058775 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.058777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.058777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.058784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.060487 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.060790 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.060791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.060791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.060863 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.061149 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.061150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.061150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.061282 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.061457 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.061522 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.061606 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 06:59:41 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
Sep 26 06:59:41 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 06:59:41.068557 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.061943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51702)
0926 06:59:41.061978 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:41.061989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:41.062026 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:41.062145 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:41.062146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:41.062290 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:41.062315 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.062316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.063023 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.063024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.063641 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.063643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.064035 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.064036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.064089 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.064411 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:41.064416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:41.065672 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:41.067586 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.067587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.067587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.067746 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.067959 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.067960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.067960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.068011 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.068252 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:41.068253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:41.068288 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 06:59:41.068355 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:59:41.068449 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 06:59:41.068603 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.062012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.1.74.124:43781)
0926 06:59:41.062044 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.062045 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.062052 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:59:41.062068 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.062069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.062077 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.062139 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.062400 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.062931 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.063712 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.063972 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.064021 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.065758 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.065758 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.065759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.065765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.067489 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.067835 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.067836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.067836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.067884 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.068080 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.068081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.068082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.068187 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.068260 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.068380 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.068495 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":15}
I20260926 06:59:41.069795 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.062374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.1.74.125:35661)
0926 06:59:41.062415 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.062418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.062461 (+ 43us) server_negotiation.cc:407] Connection header received
0926 06:59:41.062525 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.062526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.062534 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.062581 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.062996 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.063557 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.064479 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.064712 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.064772 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.066901 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.066902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.066903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.066910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.068278 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.068996 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.068997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.068998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.069058 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.069278 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.069279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.069280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.069385 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.069579 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.069617 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.069685 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:59:41.069979 14805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05: Bootstrap starting.
I20260926 06:59:41.070758 14655 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } attempt: 2
I20260926 06:59:41.070856 14805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:41.071177 14805 log.cc:824] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:41.072343 14805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05: No bootstrap required, opened a new log
I20260926 06:59:41.072781 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:59:41.074448 14805 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } }
I20260926 06:59:41.074621 14805 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:41.074667 14805 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d2205286a842b0a9578edaeffa6b05, State: Initialized, Role: FOLLOWER
I20260926 06:59:41.074795 14805 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } }
I20260926 06:59:41.075145 14805 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 06:59:41 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
I20260926 06:59:41.077803 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.077927 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:59:41.081506 14830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:59:41.081562 14830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:59:41.081919 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.074045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.1.74.125:35661)
0926 06:59:41.074083 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.074085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.074316 (+ 231us) server_negotiation.cc:407] Connection header received
0926 06:59:41.074410 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.074410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.074420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.074448 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.075216 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.075967 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.077098 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.077415 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.077459 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.078865 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.078866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.078867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.078873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.080603 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.081026 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.081027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.081028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.081083 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.081291 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.081292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.081292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.081412 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.081497 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.081726 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.081807 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:59:41.077240 14737 master_runner.cc:186] Error getting master registration for 127.1.74.126:45759: OK, instance_id { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" instance_seqno: 1790405980727901 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:59:41.082499 14737 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:59:41.084180 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.076238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.1.74.126:45759)
0926 06:59:41.076277 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.076278 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.076516 (+ 238us) server_negotiation.cc:407] Connection header received
0926 06:59:41.076798 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.076798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.076808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.076837 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.077548 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.078201 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.079207 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.079541 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.079586 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.080976 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.080977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.080977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.080985 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.082883 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.083328 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.083330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.083330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.083416 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.083613 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.083614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.083614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.083734 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.083813 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.084019 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.084094 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:59:41.084393 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.070994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51716)
0926 06:59:41.071468 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:41.071478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:41.071549 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:41.071700 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:41.071701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:41.071879 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:41.071916 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.071918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.072986 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.072987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.073715 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.073717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.074178 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.074180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.074243 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.074691 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:41.074696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:41.076279 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:41.083312 (+ 7033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.083313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.083313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.083534 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.083747 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.083748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.083748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.083808 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.084082 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:41.084083 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:41.084129 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 06:59:41.084193 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:59:41.084295 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458}
I20260926 06:59:41.084667 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.071033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.1.74.124:43781)
0926 06:59:41.071070 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.071071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.071535 (+ 464us) server_negotiation.cc:407] Connection header received
0926 06:59:41.071608 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.071609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.071619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.072089 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.072117 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.072887 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.073831 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.074113 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.074182 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.076378 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.076379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.076379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.076386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.083215 (+ 6829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.083615 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.083616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.083617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.083667 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.083876 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.083877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.083877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.083996 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.084055 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.084501 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.084575 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:59:41.085552 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.074242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.1.74.124:43781)
0926 06:59:41.074491 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.074493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.074785 (+ 292us) server_negotiation.cc:407] Connection header received
0926 06:59:41.074877 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.074878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.074890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.075032 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.075838 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.077185 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.078212 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.078527 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.078583 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.080157 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.080159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.080159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.080167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.083785 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.084133 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.084134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.084134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.084185 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.084956 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.084957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.084957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.085066 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.085180 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.085289 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.085363 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":71,"threads_started":1}
I20260926 06:59:41.086099 14655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e: Bootstrap starting.
I20260926 06:59:41.086807 14655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:41.087116 14655 log.cc:824] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:41.087946 14655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e: No bootstrap required, opened a new log
I20260926 06:59:41.089954 14655 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } }
I20260926 06:59:41.090094 14655 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:41.090129 14655 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9919b0f69d4d55ae2fc43ad8d2070e, State: Initialized, Role: FOLLOWER
I20260926 06:59:41.090268 14655 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } }
I20260926 06:59:41.090545 14833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.090667 14833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:59:41.090586 14655 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [sys.catalog]: configured and running, proceeding with master startup.
W20260926 06:59:41.092700 14844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:59:41.092767 14844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:59:41.093384 14587 master_runner.cc:186] Error getting master registration for 127.1.74.125:35661: OK, instance_id { permanent_uuid: "86d88156fd684684ba84da946582f495" instance_seqno: 1790405980877662 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:59:41.094942 14587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:59:41.120164 14730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } attempt: 3
Sep 26 06:59:41 dist-test-slave-sfvt krb5kdc[14574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
I20260926 06:59:41.128603 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.120482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:43781 (local address 127.0.0.1:51728)
0926 06:59:41.120535 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:41.120546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:41.120581 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:41.120844 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:41.120845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:41.121041 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:41.121114 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.121117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.122096 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.122097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.122936 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:41.122938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.123450 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.123451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:41.123513 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.124023 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:41.124028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:41.125557 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:41.127562 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.127563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.127564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.127757 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.127951 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:41.127951 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:41.127952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:41.128005 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:41.128293 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:41.128294 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:41.128338 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 06:59:41.128400 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:59:41.128496 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 06:59:41.129318 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:41.120659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.1.74.124:43781)
0926 06:59:41.120720 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:41.120723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:41.120731 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:59:41.120753 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:41.120754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:41.120765 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:41.120828 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:41.121206 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.122013 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.123023 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:41.123372 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:41.123444 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:41.125649 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:41.125650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:41.125651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:41.125658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:41.127442 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.127833 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.127834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.127834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.127888 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:41.128065 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:41.128066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:41.128066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:41.128200 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:41.128268 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:41.129136 (+ 868us) server_negotiation.cc:299] Negotiation successful
0926 06:59:41.129218 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:59:41.131036 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495: Bootstrap starting.
I20260926 06:59:41.131821 14730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:41.132126 14730 log.cc:824] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:41.133030 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495: No bootstrap required, opened a new log
I20260926 06:59:41.135084 14730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } }
I20260926 06:59:41.135227 14730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:41.135288 14730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d88156fd684684ba84da946582f495, State: Initialized, Role: FOLLOWER
I20260926 06:59:41.135460 14730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } }
I20260926 06:59:41.135797 14730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:59:41.135782 14847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.136022 14847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:59:41.138036 14858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:59:41.138139 14858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:59:41.140151 14662 master_runner.cc:263] No leader master found from master 86d88156fd684684ba84da946582f495
I20260926 06:59:41.140203 14662 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:59:41.171800 14847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:59:41.171976 14847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } }
I20260926 06:59:41.172243 14847 leader_election.cc:288] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9919b0f69d4d55ae2fc43ad8d2070e (127.1.74.126:45759), d1d2205286a842b0a9578edaeffa6b05 (127.1.74.124:43781)
I20260926 06:59:41.172752 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86d88156fd684684ba84da946582f495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" is_pre_election: true
I20260926 06:59:41.172936 14629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86d88156fd684684ba84da946582f495 in term 0.
I20260926 06:59:41.173404 14678 leader_election.cc:302] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86d88156fd684684ba84da946582f495, ac9919b0f69d4d55ae2fc43ad8d2070e; no voters:
I20260926 06:59:41.173779 14779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86d88156fd684684ba84da946582f495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d2205286a842b0a9578edaeffa6b05" is_pre_election: true
I20260926 06:59:41.173945 14779 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86d88156fd684684ba84da946582f495 in term 0.
I20260926 06:59:41.174222 14847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:59:41.174309 14847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:59:41.174335 14847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:41.176750 14847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } }
I20260926 06:59:41.176918 14847 leader_election.cc:288] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [CANDIDATE]: Term 1 election: Requested vote from peers ac9919b0f69d4d55ae2fc43ad8d2070e (127.1.74.126:45759), d1d2205286a842b0a9578edaeffa6b05 (127.1.74.124:43781)
I20260926 06:59:41.177781 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86d88156fd684684ba84da946582f495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e"
I20260926 06:59:41.177888 14629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:41.178846 14629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86d88156fd684684ba84da946582f495 in term 1.
I20260926 06:59:41.179118 14678 leader_election.cc:302] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86d88156fd684684ba84da946582f495, ac9919b0f69d4d55ae2fc43ad8d2070e; no voters:
I20260926 06:59:41.179394 14779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86d88156fd684684ba84da946582f495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d2205286a842b0a9578edaeffa6b05"
I20260926 06:59:41.179486 14779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:41.180415 14779 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86d88156fd684684ba84da946582f495 in term 1.
I20260926 06:59:41.180680 14847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:59:41.180864 14847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [term 1 LEADER]: Becoming Leader. State: Replica: 86d88156fd684684ba84da946582f495, State: Running, Role: LEADER
I20260926 06:59:41.180972 14847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } }
I20260926 06:59:41.181411 14847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86d88156fd684684ba84da946582f495. Latest consensus state: current_term: 1 leader_uuid: "86d88156fd684684ba84da946582f495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.181484 14847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:41.181977 14862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:59:41.182224 14862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:59:41.184401 14779 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [term 1 FOLLOWER]: Refusing update from remote peer 86d88156fd684684ba84da946582f495: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:59:41.184788 14847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:59:41.186160 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86d88156fd684684ba84da946582f495. Latest consensus state: current_term: 1 leader_uuid: "86d88156fd684684ba84da946582f495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.186268 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:59:41.186735 14629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [term 1 FOLLOWER]: Refusing update from remote peer 86d88156fd684684ba84da946582f495: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:59:41.187054 14847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:59:41.188315 14833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86d88156fd684684ba84da946582f495. Latest consensus state: current_term: 1 leader_uuid: "86d88156fd684684ba84da946582f495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.188427 14833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:59:41.189488 14860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86d88156fd684684ba84da946582f495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.189580 14860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:41.190033 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86d88156fd684684ba84da946582f495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.190117 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:59:41.190361 14863 mvcc.cc:205] Tried to move back new op lower bound from 7333502898928635904 to 7333502898917781504. Current Snapshot: MvccSnapshot[applied={T|T < 7333502898928635904}]
I20260926 06:59:41.191110 14864 mvcc.cc:205] Tried to move back new op lower bound from 7333502898928635904 to 7333502898917781504. Current Snapshot: MvccSnapshot[applied={T|T < 7333502898928635904}]
I20260926 06:59:41.191856 14862 catalog_manager.cc:1347] Generated new cluster ID: 710cff6e969e47538e302cc52b12205c
I20260926 06:59:41.191931 14862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:59:41.196502 14847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86d88156fd684684ba84da946582f495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.196593 14847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:41.197414 14833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86d88156fd684684ba84da946582f495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9919b0f69d4d55ae2fc43ad8d2070e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45759 } } peers { permanent_uuid: "86d88156fd684684ba84da946582f495" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 35661 } } peers { permanent_uuid: "d1d2205286a842b0a9578edaeffa6b05" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 43781 } } }
I20260926 06:59:41.197507 14833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:59:41.203343 14866 mvcc.cc:205] Tried to move back new op lower bound from 7333502898928635904 to 7333502898917781504. Current Snapshot: MvccSnapshot[applied={T|T < 7333502898928635904}]
I20260926 06:59:41.209499 14862 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:59:41.211198 14862 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:59:41.226995 14862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86d88156fd684684ba84da946582f495: Generated new TSK 0
I20260926 06:59:41.227326 14862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:59:42.082047 14830 catalog_manager.cc:1259] Loaded cluster ID: 710cff6e969e47538e302cc52b12205c
I20260926 06:59:42.082123 14830 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05: loading cluster ID for follower catalog manager: success
I20260926 06:59:42.082994 14830 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05: acquiring CA information for follower catalog manager: success
I20260926 06:59:42.083333 14830 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1d2205286a842b0a9578edaeffa6b05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:59:42.093158 14844 catalog_manager.cc:1259] Loaded cluster ID: 710cff6e969e47538e302cc52b12205c
I20260926 06:59:42.093216 14844 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e: loading cluster ID for follower catalog manager: success
I20260926 06:59:42.094039 14844 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e: acquiring CA information for follower catalog manager: success
I20260926 06:59:42.094357 14844 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac9919b0f69d4d55ae2fc43ad8d2070e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:59:42.276067 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 14587
I20260926 06:59:42.281219 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 14662
I20260926 06:59:42.285804 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 14737
2026-09-26T06:59:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2852 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:59:42 dist-test-slave-sfvt krb5kdc[14879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:59:42 dist-test-slave-sfvt krb5kdc[14879](info): set up 2 sockets
Sep 26 06:59:42 dist-test-slave-sfvt krb5kdc[14879](info): commencing operation
krb5kdc: starting...
W20260926 06:59:43.319278 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:59:43 dist-test-slave-sfvt krb5kdc[14879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:59:43.438535 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:44633
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:43.518738 14892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:43.518983 14892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:43.519032 14892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:43.520239 14892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:59:43.520310 14892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:43.520334 14892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:43.520363 14892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:59:43.520395 14892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:59:43.521847 14892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:43.522132 14892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:43.522360 14892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:43.527338 14900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:43.527359 14898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:43.527379 14892 server_base.cc:1034] running on GCE node
W20260926 06:59:43.527347 14897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:43.527817 14892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:43.528069 14892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:43.529210 14892 hybrid_clock.cc:648] HybridClock initialized: now 1790405983529195 us; error 34 us; skew 500 ppm
Sep 26 06:59:43 dist-test-slave-sfvt krb5kdc[14879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405983, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:43.531605 14892 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:59:43.532058 14892 webserver.cc:460] Webserver started at http://127.1.74.126:34447/ using document root <none> and password file <none>
I20260926 06:59:43.532243 14892 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:43.532282 14892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:43.532369 14892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:59:43.533367 14892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "a8ee3b97a3624a06900ed993ca684944"
format_stamp: "Formatted at 2026-09-26 06:59:43 on dist-test-slave-sfvt"
server_key: "ec170c40cfcfa3583407a60daeae7af4"
server_key_iv: "aef14d0175b286ce22a96357dc64289c"
server_key_version: "encryptionkey@0"
I20260926 06:59:43.533671 14892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "a8ee3b97a3624a06900ed993ca684944"
format_stamp: "Formatted at 2026-09-26 06:59:43 on dist-test-slave-sfvt"
server_key: "ec170c40cfcfa3583407a60daeae7af4"
server_key_iv: "aef14d0175b286ce22a96357dc64289c"
server_key_version: "encryptionkey@0"
I20260926 06:59:43.534907 14892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:59:43.535708 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:43.535879 14892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:59:43.535979 14892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "a8ee3b97a3624a06900ed993ca684944"
format_stamp: "Formatted at 2026-09-26 06:59:43 on dist-test-slave-sfvt"
server_key: "ec170c40cfcfa3583407a60daeae7af4"
server_key_iv: "aef14d0175b286ce22a96357dc64289c"
server_key_version: "encryptionkey@0"
I20260926 06:59:43.536073 14892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:43.550002 14892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:43.551858 14892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:43.552026 14892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:43.556150 14892 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:44633
I20260926 06:59:43.556182 14959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:44633 every 8 connection(s)
I20260926 06:59:43.556520 14892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:59:43.557377 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:43.559767 14960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944: Bootstrap starting.
I20260926 06:59:43.560331 14960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:43.560616 14960 log.cc:824] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:43.561342 14960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944: No bootstrap required, opened a new log
I20260926 06:59:43.562952 14960 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ee3b97a3624a06900ed993ca684944" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44633 } }
I20260926 06:59:43.563056 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 14892
I20260926 06:59:43.563107 14960 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:43.563145 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:59:43.563169 14960 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ee3b97a3624a06900ed993ca684944, State: Initialized, Role: FOLLOWER
I20260926 06:59:43.563369 1321 external_mini_cluster.cc:1428] Setting key c63d266ae5e589721e2d8c27848450de
I20260926 06:59:43.563339 14960 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ee3b97a3624a06900ed993ca684944" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44633 } }
I20260926 06:59:43.563413 14960 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:59:43.563444 14960 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:59:43.563496 14960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:43.564577 14960 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ee3b97a3624a06900ed993ca684944" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44633 } }
I20260926 06:59:43.564694 14960 leader_election.cc:302] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8ee3b97a3624a06900ed993ca684944; no voters:
I20260926 06:59:43.564870 14960 leader_election.cc:288] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:59:43.564921 14965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:59:43.565092 14965 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [term 1 LEADER]: Becoming Leader. State: Replica: a8ee3b97a3624a06900ed993ca684944, State: Running, Role: LEADER
I20260926 06:59:43.565140 14960 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:59:43.565197 14965 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ee3b97a3624a06900ed993ca684944" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44633 } }
I20260926 06:59:43.566305 14966 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8ee3b97a3624a06900ed993ca684944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ee3b97a3624a06900ed993ca684944" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44633 } } }
W20260926 06:59:43.566879 14980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 06:59:43.566939 14966 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:43.566326 14967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8ee3b97a3624a06900ed993ca684944. Latest consensus state: current_term: 1 leader_uuid: "a8ee3b97a3624a06900ed993ca684944" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ee3b97a3624a06900ed993ca684944" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44633 } } }
I20260926 06:59:43.567068 14967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944 [sys.catalog]: This master's current role is: LEADER
W20260926 06:59:43.566990 14980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:59:43.567373 14982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:59:43.567576 14982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:59:43.569362 14982 catalog_manager.cc:1347] Generated new cluster ID: 2089c01ac35a4305b06c6cf8250ec1c2
I20260926 06:59:43.569411 14982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:59:43 dist-test-slave-sfvt krb5kdc[14879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:43.575647 14964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:43.563893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.1.74.126:44633)
0926 06:59:43.564088 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:43.564096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:43.564121 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:59:43.564211 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:43.564217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:43.564256 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:43.564409 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:43.565253 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:43.566779 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:43.567926 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:43.568527 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:43.568663 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:43.571481 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:43.571485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:43.571486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:43.571497 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:43.573926 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:43.574331 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:43.574334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:43.574335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:43.574451 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:43.574667 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:43.574667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:43.574668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:43.574953 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:43.575092 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:43.575245 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 06:59:43.575407 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
I20260926 06:59:43.576552 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:59:43.582283 14982 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:59:43.582804 14982 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:59:43.587424 14982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8ee3b97a3624a06900ed993ca684944: Generated new TSK 0
I20260926 06:59:43.587612 14982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:59:43.630419 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 14892
2026-09-26T06:59:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1343 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:59:43 dist-test-slave-sfvt krb5kdc[14987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:59:43 dist-test-slave-sfvt krb5kdc[14987](info): set up 2 sockets
Sep 26 06:59:43 dist-test-slave-sfvt krb5kdc[14987](info): commencing operation
krb5kdc: starting...
W20260926 06:59:44.675390 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 06:59:44 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:59:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:59:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 06:59:44.790208 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42169
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:42169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:44.870641 15000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:44.870837 15000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:44.870858 15000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:44.871994 15000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:59:44.872032 15000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:44.872046 15000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:44.872059 15000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:59:44.872071 15000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:59:44.873432 15000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:42169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:44.873611 15000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:44.873769 15000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:44.878546 15006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:44.878743 15000 server_base.cc:1034] running on GCE node
W20260926 06:59:44.878554 15008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:44.878633 15005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:44.879087 15000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:44.879416 15000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:44.880573 15000 hybrid_clock.cc:648] HybridClock initialized: now 1790405984880570 us; error 34 us; skew 500 ppm
Sep 26 06:59:44 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405984, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:44.882977 15000 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:59:44.883509 15000 webserver.cc:460] Webserver started at http://127.1.74.126:33061/ using document root <none> and password file <none>
I20260926 06:59:44.883728 15000 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:44.883795 15000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:44.883957 15000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:59:44.884971 15000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "ecd8cbf76b404434af5fbe95c6fc83c3"
format_stamp: "Formatted at 2026-09-26 06:59:44 on dist-test-slave-sfvt"
server_key: "002b10368d5af72c3368f4692891ecf5"
server_key_iv: "bd5587393f779016adead3bc299dee20"
server_key_version: "encryptionkey@0"
I20260926 06:59:44.885340 15000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "ecd8cbf76b404434af5fbe95c6fc83c3"
format_stamp: "Formatted at 2026-09-26 06:59:44 on dist-test-slave-sfvt"
server_key: "002b10368d5af72c3368f4692891ecf5"
server_key_iv: "bd5587393f779016adead3bc299dee20"
server_key_version: "encryptionkey@0"
I20260926 06:59:44.886736 15000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:59:44.887545 15015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:44.887727 15000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:44.887825 15000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "ecd8cbf76b404434af5fbe95c6fc83c3"
format_stamp: "Formatted at 2026-09-26 06:59:44 on dist-test-slave-sfvt"
server_key: "002b10368d5af72c3368f4692891ecf5"
server_key_iv: "bd5587393f779016adead3bc299dee20"
server_key_version: "encryptionkey@0"
I20260926 06:59:44.887907 15000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:44.907735 15000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:44.910108 15000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:44.910282 15000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:44.914362 15000 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:42169
I20260926 06:59:44.914400 15067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:42169 every 8 connection(s)
I20260926 06:59:44.914796 15000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:59:44.915853 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:44.918071 15068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Bootstrap starting.
I20260926 06:59:44.918651 15068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:44.918929 15068 log.cc:824] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:44.919649 15068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: No bootstrap required, opened a new log
I20260926 06:59:44.921188 15068 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:44.921327 15068 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:44.921350 15068 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd8cbf76b404434af5fbe95c6fc83c3, State: Initialized, Role: FOLLOWER
I20260926 06:59:44.921450 15068 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:44.921532 15068 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:59:44.921563 15068 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:59:44.921615 15068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:44.922407 15068 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:44.922514 15068 leader_election.cc:302] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecd8cbf76b404434af5fbe95c6fc83c3; no voters:
I20260926 06:59:44.922685 15068 leader_election.cc:288] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:59:44.922732 15071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:59:44.922897 15071 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 1 LEADER]: Becoming Leader. State: Replica: ecd8cbf76b404434af5fbe95c6fc83c3, State: Running, Role: LEADER
I20260926 06:59:44.922941 15068 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:59:44.923010 15071 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:44.923401 15072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } } }
I20260926 06:59:44.923414 15073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecd8cbf76b404434af5fbe95c6fc83c3. Latest consensus state: current_term: 1 leader_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } } }
I20260926 06:59:44.923565 15073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:44.923770 15072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:44.924017 15080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:59:44.924528 15080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:59:44.924911 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 15000
I20260926 06:59:44.925007 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 06:59:44.925194 1321 external_mini_cluster.cc:1428] Setting key 2a013a1ca770dd061942de4302bbc6df
I20260926 06:59:44.926774 15080 catalog_manager.cc:1347] Generated new cluster ID: cc792b77d7fb41d3915cab7f1f29de0c
I20260926 06:59:44.926831 15080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:59:44 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:44.935039 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:44.925631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.1.74.126:42169)
0926 06:59:44.925798 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:44.925808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:44.925926 (+ 118us) server_negotiation.cc:407] Connection header received
0926 06:59:44.926085 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:44.926090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:44.926161 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:44.926303 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:44.927161 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:44.928124 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:44.928883 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:44.929349 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:44.929463 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:44.931785 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:44.931788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:44.931789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:44.931800 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:44.933485 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:44.933858 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:44.933861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:44.933862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:44.933961 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:44.934180 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:44.934181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:44.934181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:44.934415 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:44.934574 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:44.934678 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 06:59:44.934832 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260926 06:59:44.939430 15080 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:59:44.940140 15080 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:59:44.947674 15080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Generated new TSK 0
I20260926 06:59:44.947860 15080 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 06:59:44.976235 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42169
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:45.056191 15094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:45.056391 15094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:45.056412 15094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:45.057619 15094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:45.057662 15094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:45.057691 15094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:59:45.059127 15094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:45.059397 15094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:45.059633 15094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:45.064656 15099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:45.064631 15100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:45.064636 15102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.064723 15094 server_base.cc:1034] running on GCE node
I20260926 06:59:45.065151 15094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:45.065451 15094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:45.066594 15094 hybrid_clock.cc:648] HybridClock initialized: now 1790405985066570 us; error 40 us; skew 500 ppm
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:45.069029 15094 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:59:45.069470 15094 webserver.cc:460] Webserver started at http://127.1.74.65:46185/ using document root <none> and password file <none>
I20260926 06:59:45.069693 15094 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:45.069759 15094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:45.069912 15094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:59:45.070959 15094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "444edb1aa920424395882c1f96d49299"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "af48b5ad833413bfcc150a73c1748f58"
server_key_iv: "2b1a41a86c60158059c560c23c79254c"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.071399 15094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "444edb1aa920424395882c1f96d49299"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "af48b5ad833413bfcc150a73c1748f58"
server_key_iv: "2b1a41a86c60158059c560c23c79254c"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.072726 15094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:45.073417 15109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.073570 15094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:59:45.073665 15094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "444edb1aa920424395882c1f96d49299"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "af48b5ad833413bfcc150a73c1748f58"
server_key_iv: "2b1a41a86c60158059c560c23c79254c"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.073750 15094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:45.098233 15094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:45.100314 15094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:45.100483 15094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:45.100723 15094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:45.101038 15094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:45.101094 15094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.101135 15094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:45.101163 15094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.106910 15094 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:44825
I20260926 06:59:45.106959 15222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:44825 every 8 connection(s)
I20260926 06:59:45.107327 15094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:59:45.111114 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 15094
I20260926 06:59:45.111239 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 06:59:45.111433 1321 external_mini_cluster.cc:1428] Setting key 85629f87a91e3995e63f2059eb5ea572
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:45.117089 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.108066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:59121 (local address 127.1.74.126:42169)
0926 06:59:45.108152 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.108154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.108317 (+ 163us) server_negotiation.cc:407] Connection header received
0926 06:59:45.108769 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.108770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.108787 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.108837 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:45.109699 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.110219 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.111206 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.111484 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.111565 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.114037 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:45.114039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:45.114039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:45.114047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:45.115588 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.116082 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.116082 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.116083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.116133 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.116417 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.116418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.116418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.116520 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:45.116569 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.116939 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.117014 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:59:45.117401 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.108015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:59121)
0926 06:59:45.108191 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.108227 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.108680 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.108889 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.108892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.109436 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:45.109624 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.109628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.110296 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.110296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.111102 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.111104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.111595 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.111597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.111748 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.112293 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:45.112304 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:45.113936 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:45.115665 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.115673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.115676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.116006 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.116197 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.116198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.116199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.116345 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.116587 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:45.116595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:45.116791 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.116923 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.117120 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260926 06:59:45.118008 15223 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42169
I20260926 06:59:45.118106 15223 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:45.118304 15223 heartbeater.cc:511] Master 127.1.74.126:42169 requested a full tablet report, sending...
I20260926 06:59:45.118824 15032 ts_manager.cc:194] Registered new tserver with Master: 444edb1aa920424395882c1f96d49299 (127.1.74.65:44825)
I20260926 06:59:45.119568 15032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:59121
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 06:59:45.138481 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42169
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:45.219956 15228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:45.220175 15228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:45.220198 15228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:45.221360 15228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:45.221405 15228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:45.221432 15228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:59:45.222954 15228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:45.223192 15228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:45.223510 15228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:45.228760 15233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:45.228736 15234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:45.228751 15236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.228902 15228 server_base.cc:1034] running on GCE node
I20260926 06:59:45.229319 15228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:45.229560 15228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:45.230702 15228 hybrid_clock.cc:648] HybridClock initialized: now 1790405985230688 us; error 31 us; skew 500 ppm
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:45.233357 15228 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:59:45.233777 15228 webserver.cc:460] Webserver started at http://127.1.74.66:41309/ using document root <none> and password file <none>
I20260926 06:59:45.234006 15228 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:45.234051 15228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:45.234196 15228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:59:45.235232 15228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "2182d3ae26574f9caedcdf253178c5cd"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "88d05f99dbe546331da9771cd83bd25f"
server_key_iv: "3523cfc1290a6d89d96055b96759b7d2"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.235684 15228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "2182d3ae26574f9caedcdf253178c5cd"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "88d05f99dbe546331da9771cd83bd25f"
server_key_iv: "3523cfc1290a6d89d96055b96759b7d2"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.236969 15228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:45.237677 15243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.237869 15228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:59:45.237958 15228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "2182d3ae26574f9caedcdf253178c5cd"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "88d05f99dbe546331da9771cd83bd25f"
server_key_iv: "3523cfc1290a6d89d96055b96759b7d2"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.238034 15228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:45.252242 15228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:45.254312 15228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:45.254462 15228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:45.254724 15228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:45.255059 15228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:45.255115 15228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.255147 15228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:45.255178 15228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.261337 15228 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:40191
I20260926 06:59:45.261394 15356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:40191 every 8 connection(s)
I20260926 06:59:45.261703 15228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:59:45.263381 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 15228
I20260926 06:59:45.263459 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 06:59:45.263650 1321 external_mini_cluster.cc:1428] Setting key a2fa75b3f1cf6c1937835d36f211f875
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
I20260926 06:59:45.272755 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.262417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:36743 (local address 127.1.74.126:42169)
0926 06:59:45.262477 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.262479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.262691 (+ 212us) server_negotiation.cc:407] Connection header received
0926 06:59:45.263233 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.263234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.263247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.263338 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:45.264538 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.265045 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.266500 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.266693 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.266758 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.269323 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:45.269325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:45.269325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:45.269335 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:45.271080 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.271643 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.271645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.271645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.271711 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.272006 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.272007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.272008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.272168 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:45.272244 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.272535 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.272635 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:59:45.272997 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.262392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:36743)
0926 06:59:45.262546 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.262589 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.263125 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.263420 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.263424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.264174 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:45.264442 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.264449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.265149 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.265150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.266405 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.266408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.266776 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.266777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.266931 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.267514 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:45.267525 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:45.269216 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:45.271194 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.271202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.271204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.271548 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.271798 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.271798 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.271799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.271923 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.272249 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:45.272254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:45.272388 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.272490 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.272696 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260926 06:59:45.273506 15357 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42169
I20260926 06:59:45.273684 15357 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:45.273918 15357 heartbeater.cc:511] Master 127.1.74.126:42169 requested a full tablet report, sending...
I20260926 06:59:45.274358 15032 ts_manager.cc:194] Registered new tserver with Master: 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191)
I20260926 06:59:45.274900 15032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:36743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 06:59:45.291173 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42169
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:45.372294 15362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:45.372502 15362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:45.372525 15362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:45.373673 15362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:45.373716 15362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:45.373744 15362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:59:45.375248 15362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:45.375516 15362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:45.375741 15362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:45.380825 15370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.380990 15362 server_base.cc:1034] running on GCE node
W20260926 06:59:45.380870 15368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:45.380834 15367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.381304 15362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:45.381511 15362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:45.382644 15362 hybrid_clock.cc:648] HybridClock initialized: now 1790405985382632 us; error 28 us; skew 500 ppm
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:45.385061 15362 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:59:45.385514 15362 webserver.cc:460] Webserver started at http://127.1.74.67:46739/ using document root <none> and password file <none>
I20260926 06:59:45.385733 15362 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:45.385800 15362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:45.385950 15362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:59:45.386973 15362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "a5027532a86d4155b0c79e89dc7bde51"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "52bd42f294ebac4da136d6b81243fb75"
server_key_iv: "4923a82b156b18bcb19af09bd3cfec7d"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.387418 15362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "a5027532a86d4155b0c79e89dc7bde51"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "52bd42f294ebac4da136d6b81243fb75"
server_key_iv: "4923a82b156b18bcb19af09bd3cfec7d"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.388726 15362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:45.389439 15377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.389642 15362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:59:45.389735 15362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "a5027532a86d4155b0c79e89dc7bde51"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "52bd42f294ebac4da136d6b81243fb75"
server_key_iv: "4923a82b156b18bcb19af09bd3cfec7d"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.389823 15362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:45.415321 15362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:45.417297 15362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:45.417469 15362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:45.417711 15362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:45.418027 15362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:45.418082 15362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.418125 15362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:45.418151 15362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.424208 15362 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:45523
I20260926 06:59:45.424279 15490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:45523 every 8 connection(s)
I20260926 06:59:45.424638 15362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:59:45.425956 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 15362
I20260926 06:59:45.426052 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 06:59:45.426214 1321 external_mini_cluster.cc:1428] Setting key 789768d8bec186678b1cfc923869d15f
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:45.433898 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.425433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:34933 (local address 127.1.74.126:42169)
0926 06:59:45.425476 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.425478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.425697 (+ 219us) server_negotiation.cc:407] Connection header received
0926 06:59:45.426402 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.426404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.426418 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.426484 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:45.427669 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.428157 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.429100 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.429275 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.429330 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.431401 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:45.431402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:45.431403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:45.431409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:45.432814 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.433155 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.433156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.433157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.433211 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.433412 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.433412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.433413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.433509 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:45.433558 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.433776 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.433834 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:59:45.434077 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.425427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:34933)
0926 06:59:45.425572 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.425610 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.426295 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.426570 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.426575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.427384 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:45.427588 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.427592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.428234 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.428235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.429028 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.429030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.429342 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.429343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.429439 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.429902 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:45.429910 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:45.431315 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:45.432884 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.432889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.432891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.433093 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.433267 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.433268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.433268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.433359 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.433562 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:45.433566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:45.433673 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.433750 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.433862 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20260926 06:59:45.434564 15491 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42169
I20260926 06:59:45.434675 15491 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:45.434857 15491 heartbeater.cc:511] Master 127.1.74.126:42169 requested a full tablet report, sending...
I20260926 06:59:45.435325 15032 ts_manager.cc:194] Registered new tserver with Master: a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523)
I20260926 06:59:45.435719 15032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:34933
I20260926 06:59:45.437707 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:59:45.444558 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.438978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.1.74.126:42169)
0926 06:59:45.439016 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.439018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.439163 (+ 145us) server_negotiation.cc:407] Connection header received
0926 06:59:45.439180 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.439180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.439188 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.439224 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:45.439819 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.440337 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.441080 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.441270 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.441325 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.442448 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:45.442449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:45.442449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:45.442454 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:45.443657 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.443941 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.443942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.443943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.443987 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.444139 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.444140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.444140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.444270 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:45.444341 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.444446 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.444500 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:59:45.445991 15032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:59:45.447687 15032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:59:45.454255 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.450244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.1.74.65:44825)
0926 06:59:45.450401 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.450405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.450682 (+ 277us) server_negotiation.cc:407] Connection header received
0926 06:59:45.450730 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.450733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.450749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.450828 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:45.451026 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.451788 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.453260 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.453825 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.453923 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.453985 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.454138 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.454179 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260926 06:59:45.454247 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.450096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44825 (local address 127.0.0.1:58442)
0926 06:59:45.450578 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.450585 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.450625 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.450907 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.450908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.450911 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:45.450963 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.450964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.451880 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.451881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.453169 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.453171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.453927 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.453928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.454004 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.454017 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.454119 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.454174 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":29,"thread_start_us":73,"threads_started":1}
I20260926 06:59:45.454484 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.449901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:45523 (local address 127.0.0.1:39956)
0926 06:59:45.450402 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.450413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.450475 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.450770 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.450773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.450774 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:45.450857 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.450859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.452428 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.452429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.453353 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.453356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.454235 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.454236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.454302 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.454311 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.454361 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.454415 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":12,"thread_start_us":59,"threads_started":1}
I20260926 06:59:45.454566 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.449999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.1.74.66:40191)
0926 06:59:45.450195 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.450201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.450479 (+ 278us) server_negotiation.cc:407] Connection header received
0926 06:59:45.450565 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.450567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.450603 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.450674 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:45.450925 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.452054 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.453350 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.454067 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.454143 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.454198 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.454429 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.454470 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260926 06:59:45.454763 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.449901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:40191 (local address 127.0.0.1:48800)
0926 06:59:45.450348 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.450389 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.450475 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.450768 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.450771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.450773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:45.450856 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.450859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.452168 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.452169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.453229 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.453231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.454241 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.454242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.454302 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.454312 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.454659 (+ 347us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.454703 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":11,"thread_start_us":57,"threads_started":1}
I20260926 06:59:45.454792 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.450063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.1.74.67:45523)
0926 06:59:45.450269 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.450274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.450478 (+ 204us) server_negotiation.cc:407] Connection header received
0926 06:59:45.450565 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.450567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.450585 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.450671 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:45.451340 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.452334 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.453426 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.454095 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.454152 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.454198 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.454660 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.454704 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260926 06:59:45.455380 15157 tablet_service.cc:1467] Processing CreateTablet for tablet ba4e3ff7974e4eaab1df2b250d6cb20d (DEFAULT_TABLE table=test-table [id=e6e0d9a0778047ca881e2385190c07f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:59:45.455466 15425 tablet_service.cc:1467] Processing CreateTablet for tablet ba4e3ff7974e4eaab1df2b250d6cb20d (DEFAULT_TABLE table=test-table [id=e6e0d9a0778047ca881e2385190c07f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:59:45.455523 15291 tablet_service.cc:1467] Processing CreateTablet for tablet ba4e3ff7974e4eaab1df2b250d6cb20d (DEFAULT_TABLE table=test-table [id=e6e0d9a0778047ca881e2385190c07f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:59:45.455766 15157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4e3ff7974e4eaab1df2b250d6cb20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:45.455801 15425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4e3ff7974e4eaab1df2b250d6cb20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:45.455801 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4e3ff7974e4eaab1df2b250d6cb20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:45.458424 15508 tablet_bootstrap.cc:492] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Bootstrap starting.
I20260926 06:59:45.459049 15510 tablet_bootstrap.cc:492] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: Bootstrap starting.
I20260926 06:59:45.459096 15508 tablet_bootstrap.cc:654] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:45.459420 15509 tablet_bootstrap.cc:492] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: Bootstrap starting.
I20260926 06:59:45.459699 15508 log.cc:824] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:45.459867 15510 tablet_bootstrap.cc:654] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:45.460068 15509 tablet_bootstrap.cc:654] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:45.460446 15510 log.cc:824] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:45.460502 15508 tablet_bootstrap.cc:492] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: No bootstrap required, opened a new log
I20260926 06:59:45.460558 15508 ts_tablet_manager.cc:1397] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:59:45.460577 15509 log.cc:824] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:45.461225 15510 tablet_bootstrap.cc:492] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: No bootstrap required, opened a new log
I20260926 06:59:45.461305 15510 ts_tablet_manager.cc:1397] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:59:45.461328 15509 tablet_bootstrap.cc:492] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: No bootstrap required, opened a new log
I20260926 06:59:45.461428 15509 ts_tablet_manager.cc:1397] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:59:45.462244 15508 raft_consensus.cc:348] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.462419 15508 raft_consensus.cc:374] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:45.462464 15508 raft_consensus.cc:729] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444edb1aa920424395882c1f96d49299, State: Initialized, Role: FOLLOWER
I20260926 06:59:45.462576 15508 consensus_queue.cc:260] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.462872 15223 heartbeater.cc:503] Master 127.1.74.126:42169 was elected leader, sending a full tablet report...
I20260926 06:59:45.462884 15508 ts_tablet_manager.cc:1428] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:59:45.463300 15510 raft_consensus.cc:348] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.463352 15509 raft_consensus.cc:348] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.463474 15510 raft_consensus.cc:374] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:45.463487 15509 raft_consensus.cc:374] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:45.463518 15509 raft_consensus.cc:729] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2182d3ae26574f9caedcdf253178c5cd, State: Initialized, Role: FOLLOWER
I20260926 06:59:45.463518 15510 raft_consensus.cc:729] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5027532a86d4155b0c79e89dc7bde51, State: Initialized, Role: FOLLOWER
I20260926 06:59:45.463621 15510 consensus_queue.cc:260] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.463640 15509 consensus_queue.cc:260] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.463899 15509 ts_tablet_manager.cc:1428] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:59:45.463899 15510 ts_tablet_manager.cc:1428] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:59:45.464003 15491 heartbeater.cc:503] Master 127.1.74.126:42169 was elected leader, sending a full tablet report...
I20260926 06:59:45.464003 15357 heartbeater.cc:503] Master 127.1.74.126:42169 was elected leader, sending a full tablet report...
W20260926 06:59:45.512346 15358 tablet.cc:2369] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:59:45.550063 15514 raft_consensus.cc:482] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:59:45.550181 15514 raft_consensus.cc:504] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.550496 15514 leader_election.cc:288] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523), 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191)
I20260926 06:59:45.554428 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.550872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:45397 (local address 127.1.74.66:40191)
0926 06:59:45.550948 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.550950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.550960 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:59:45.550988 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.550989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.551004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.551074 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:45.551283 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.552038 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.553492 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.554062 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.554137 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.554147 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.554317 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.554355 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:59:45.554428 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.550816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:52733 (local address 127.1.74.67:45523)
0926 06:59:45.550861 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.550863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.551171 (+ 308us) server_negotiation.cc:407] Connection header received
0926 06:59:45.551195 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.551196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.551227 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.551326 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:45.551459 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.552131 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.553489 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.554062 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.554137 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.554147 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.554317 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.554355 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:59:45.554440 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.550694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:40191 (local address 127.1.74.65:45397)
0926 06:59:45.550760 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.550775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.550842 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.551089 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.551090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.551092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:45.551166 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.551168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.552117 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.552117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.553399 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.553402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.554150 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.554151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.554234 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.554236 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.554297 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.554340 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":49}
I20260926 06:59:45.554446 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.550694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:45523 (local address 127.1.74.65:52733)
0926 06:59:45.551083 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.551093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.551147 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.551335 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.551336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.551338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:45.551388 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.551389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.552200 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.552200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.553399 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.553402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.554150 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.554151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.554230 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.554235 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.554297 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.554340 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":52,"thread_start_us":284,"threads_started":1}
I20260926 06:59:45.554826 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4e3ff7974e4eaab1df2b250d6cb20d" candidate_uuid: "444edb1aa920424395882c1f96d49299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2182d3ae26574f9caedcdf253178c5cd" is_pre_election: true
I20260926 06:59:45.554858 15445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4e3ff7974e4eaab1df2b250d6cb20d" candidate_uuid: "444edb1aa920424395882c1f96d49299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5027532a86d4155b0c79e89dc7bde51" is_pre_election: true
I20260926 06:59:45.554977 15311 raft_consensus.cc:2454] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 444edb1aa920424395882c1f96d49299 in term 0.
I20260926 06:59:45.555013 15445 raft_consensus.cc:2454] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 444edb1aa920424395882c1f96d49299 in term 0.
I20260926 06:59:45.555233 15110 leader_election.cc:302] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444edb1aa920424395882c1f96d49299, a5027532a86d4155b0c79e89dc7bde51; no voters:
I20260926 06:59:45.555444 15514 raft_consensus.cc:2781] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:59:45.555542 15514 raft_consensus.cc:482] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:59:45.555578 15514 raft_consensus.cc:3037] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:45.556291 15514 raft_consensus.cc:504] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.556447 15514 leader_election.cc:288] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 1 election: Requested vote from peers a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523), 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191)
I20260926 06:59:45.556564 15445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4e3ff7974e4eaab1df2b250d6cb20d" candidate_uuid: "444edb1aa920424395882c1f96d49299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5027532a86d4155b0c79e89dc7bde51"
I20260926 06:59:45.556648 15311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4e3ff7974e4eaab1df2b250d6cb20d" candidate_uuid: "444edb1aa920424395882c1f96d49299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2182d3ae26574f9caedcdf253178c5cd"
I20260926 06:59:45.556661 15445 raft_consensus.cc:3037] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:45.556733 15311 raft_consensus.cc:3037] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:45.557390 15445 raft_consensus.cc:2454] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 444edb1aa920424395882c1f96d49299 in term 1.
I20260926 06:59:45.557410 15311 raft_consensus.cc:2454] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 444edb1aa920424395882c1f96d49299 in term 1.
I20260926 06:59:45.557626 15112 leader_election.cc:302] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2182d3ae26574f9caedcdf253178c5cd, 444edb1aa920424395882c1f96d49299; no voters:
I20260926 06:59:45.557724 15514 raft_consensus.cc:2781] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:59:45.557894 15514 raft_consensus.cc:686] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 1 LEADER]: Becoming Leader. State: Replica: 444edb1aa920424395882c1f96d49299, State: Running, Role: LEADER
I20260926 06:59:45.558002 15514 consensus_queue.cc:237] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.558881 15031 catalog_manager.cc:5495] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 reported cstate change: term changed from 0 to 1, leader changed from <none> to 444edb1aa920424395882c1f96d49299 (127.1.74.65). New cstate: current_term: 1 leader_uuid: "444edb1aa920424395882c1f96d49299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } health_report { overall_health: HEALTHY } } }
I20260926 06:59:45.570745 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.568412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.1.74.65:44825)
0926 06:59:45.568451 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.568454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.568462 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:59:45.568523 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.568524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.568553 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.568631 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:59:45.568748 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.569211 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.569955 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.570152 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.570225 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.570402 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:59:45.570503 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:59:45.570597 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.570616 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.570638 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:59:45.572374 15311 raft_consensus.cc:1261] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [term 1 FOLLOWER]: Refusing update from remote peer 444edb1aa920424395882c1f96d49299: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:59:45.572373 15445 raft_consensus.cc:1261] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [term 1 FOLLOWER]: Refusing update from remote peer 444edb1aa920424395882c1f96d49299: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:59:45.572710 15514 consensus_queue.cc:1035] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:59:45.572855 15518 consensus_queue.cc:1035] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:59:45.575049 15521 mvcc.cc:205] Tried to move back new op lower bound from 7333502916901679104 to 7333502916846002176. Current Snapshot: MvccSnapshot[applied={T|T < 7333502916901679104}]
I20260926 06:59:45.575323 15522 mvcc.cc:205] Tried to move back new op lower bound from 7333502916901679104 to 7333502916846002176. Current Snapshot: MvccSnapshot[applied={T|T < 7333502916901679104}]
I20260926 06:59:45.575861 15523 mvcc.cc:205] Tried to move back new op lower bound from 7333502916901679104 to 7333502916846002176. Current Snapshot: MvccSnapshot[applied={T|T < 7333502916901679104}]
I20260926 06:59:45.577440 15031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:59:45.577550 15031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:59:45.578289 15031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Sending DeleteTablet for 3 replicas of tablet ba4e3ff7974e4eaab1df2b250d6cb20d
I20260926 06:59:45.578614 15157 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4e3ff7974e4eaab1df2b250d6cb20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:45 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:58442
I20260926 06:59:45.578640 15425 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4e3ff7974e4eaab1df2b250d6cb20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:45 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:39956
I20260926 06:59:45.578742 15291 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4e3ff7974e4eaab1df2b250d6cb20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:45 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:48800
I20260926 06:59:45.578903 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 15094
I20260926 06:59:45.578960 15533 tablet_replica.cc:330] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: stopping tablet replica
I20260926 06:59:45.579607 15533 raft_consensus.cc:2229] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:59:45.579694 15533 raft_consensus.cc:2258] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:59:45.580062 15533 ts_tablet_manager.cc:1905] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:59:45.581440 15532 tablet_replica.cc:330] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: stopping tablet replica
I20260926 06:59:45.581557 15532 raft_consensus.cc:2229] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:59:45.581655 15532 raft_consensus.cc:2258] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:59:45.581805 15533 ts_tablet_manager.cc:1918] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:59:45.581889 15533 log.cc:1196] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/ba4e3ff7974e4eaab1df2b250d6cb20d
I20260926 06:59:45.582080 15532 ts_tablet_manager.cc:1905] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:59:45.582168 15533 ts_tablet_manager.cc:1939] T ba4e3ff7974e4eaab1df2b250d6cb20d P 2182d3ae26574f9caedcdf253178c5cd: Deleting consensus metadata
I20260926 06:59:45.582573 15018 catalog_manager.cc:4841] TS 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191): tablet ba4e3ff7974e4eaab1df2b250d6cb20d (table test-table [id=e6e0d9a0778047ca881e2385190c07f2]) successfully deleted
I20260926 06:59:45.584020 15532 ts_tablet_manager.cc:1918] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:59:45.584120 15532 log.cc:1196] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/ba4e3ff7974e4eaab1df2b250d6cb20d
I20260926 06:59:45.584380 15532 ts_tablet_manager.cc:1939] T ba4e3ff7974e4eaab1df2b250d6cb20d P a5027532a86d4155b0c79e89dc7bde51: Deleting consensus metadata
I20260926 06:59:45.584776 15016 catalog_manager.cc:4841] TS a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523): tablet ba4e3ff7974e4eaab1df2b250d6cb20d (table test-table [id=e6e0d9a0778047ca881e2385190c07f2]) successfully deleted
W20260926 06:59:45.585783 15018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:44825 (error 108)
I20260926 06:59:45.585983 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 15228
I20260926 06:59:45.586705 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.586589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44825 (local address 127.0.0.1:58462)
0926 06:59:45.586637 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.586679 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 06:59:45.586838 15018 catalog_manager.cc:4584] TS 444edb1aa920424395882c1f96d49299 (127.1.74.65:44825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba4e3ff7974e4eaab1df2b250d6cb20d: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44825: connect: Connection refused (error 111)
I20260926 06:59:45.591022 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 15362
I20260926 06:59:45.595835 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 15000
I20260926 06:59:45.601171 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42169
--webserver_interface=127.1.74.126
--webserver_port=33061
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:42169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:45.681734 15535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:45.681944 15535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:45.681967 15535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:45.683023 15535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:59:45.683056 15535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:45.683070 15535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:45.683082 15535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:59:45.683094 15535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:59:45.684474 15535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:42169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:45.684655 15535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:45.684823 15535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:45.693292 15544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:45.693308 15541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.693331 15535 server_base.cc:1034] running on GCE node
W20260926 06:59:45.693370 15542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.693914 15535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:45.694219 15535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:45.695405 15535 hybrid_clock.cc:648] HybridClock initialized: now 1790405985695396 us; error 38 us; skew 500 ppm
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:45.697777 15535 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:59:45.698227 15535 webserver.cc:460] Webserver started at http://127.1.74.126:33061/ using document root <none> and password file <none>
I20260926 06:59:45.698468 15535 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:45.698586 15535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:45.700018 15535 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:59:45.700650 15551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.700834 15535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:59:45.700927 15535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "ecd8cbf76b404434af5fbe95c6fc83c3"
format_stamp: "Formatted at 2026-09-26 06:59:44 on dist-test-slave-sfvt"
server_key: "002b10368d5af72c3368f4692891ecf5"
server_key_iv: "bd5587393f779016adead3bc299dee20"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.701336 15535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:45.716634 15535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:45.718531 15535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:45.718703 15535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:45.722916 15535 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:42169
I20260926 06:59:45.722985 15603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:42169 every 8 connection(s)
I20260926 06:59:45.723495 15535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:59:45.725088 15604 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:59:45.725793 15604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Bootstrap starting.
I20260926 06:59:45.725852 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 15535
I20260926 06:59:45.726207 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:44825
--local_ip_for_outbound_sockets=127.1.74.65
--tserver_master_addrs=127.1.74.126:42169
--webserver_port=46185
--webserver_interface=127.1.74.65
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:45.734131 15604 log.cc:824] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:45.736176 15604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:59:45.736491 15604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Bootstrap complete.
I20260926 06:59:45.738984 15604 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:45.739238 15604 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd8cbf76b404434af5fbe95c6fc83c3, State: Initialized, Role: FOLLOWER
I20260926 06:59:45.739452 15604 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:45.739565 15604 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:59:45.739642 15604 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:59:45.739693 15604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:59:45.740705 15604 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:45.740793 15604 leader_election.cc:302] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecd8cbf76b404434af5fbe95c6fc83c3; no voters:
I20260926 06:59:45.741009 15604 leader_election.cc:288] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:59:45.741047 15608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:59:45.741201 15608 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 2 LEADER]: Becoming Leader. State: Replica: ecd8cbf76b404434af5fbe95c6fc83c3, State: Running, Role: LEADER
I20260926 06:59:45.741289 15604 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:59:45.741351 15608 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:45.741699 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } } }
I20260926 06:59:45.741784 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:45.742020 15610 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecd8cbf76b404434af5fbe95c6fc83c3. Latest consensus state: current_term: 2 leader_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } } }
I20260926 06:59:45.742102 15610 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:45.742197 15615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:59:45.743045 15615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:59:45.743309 15615 catalog_manager.cc:1259] Loaded cluster ID: cc792b77d7fb41d3915cab7f1f29de0c
I20260926 06:59:45.743474 15615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:59:45.743709 15615 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 06:59:45.746413 15615 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 06:59:45.752188 15615 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:59:45.752393 15615 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Loaded TSK: 0
I20260926 06:59:45.752601 15615 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:59:45.825309 15606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:45.825562 15606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:45.825600 15606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:45.826745 15606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:45.826820 15606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:45.826867 15606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:59:45.828408 15606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:44825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=46185
--tserver_master_addrs=127.1.74.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:45.828671 15606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:45.828900 15606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:45.833894 15631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:45.833951 15632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.834115 15606 server_base.cc:1034] running on GCE node
W20260926 06:59:45.833863 15634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.834461 15606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:45.834755 15606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:45.835938 15606 hybrid_clock.cc:648] HybridClock initialized: now 1790405985835922 us; error 45 us; skew 500 ppm
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:45.838879 15606 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:59:45.839402 15606 webserver.cc:460] Webserver started at http://127.1.74.65:46185/ using document root <none> and password file <none>
I20260926 06:59:45.839648 15606 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:45.839730 15606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:45.841171 15606 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:45.842012 15641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.842180 15606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:45.842267 15606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "444edb1aa920424395882c1f96d49299"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "af48b5ad833413bfcc150a73c1748f58"
server_key_iv: "2b1a41a86c60158059c560c23c79254c"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.842660 15606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:45.858760 15606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:45.861073 15606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:45.861287 15606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:45.861564 15606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:45.862133 15648 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:59:45.864032 15606 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:59:45.864166 15606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:59:45.864233 15606 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:59:45.864928 15606 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:59:45.864987 15606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:59:45.865056 15648 tablet_bootstrap.cc:492] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Bootstrap starting.
I20260926 06:59:45.870399 15606 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:44825
I20260926 06:59:45.870792 15606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:59:45.871536 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 15606
I20260926 06:59:45.871867 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:40191
--local_ip_for_outbound_sockets=127.1.74.66
--tserver_master_addrs=127.1.74.126:42169
--webserver_port=41309
--webserver_interface=127.1.74.66
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:45.878813 15648 log.cc:824] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:45.880544 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:44825 every 8 connection(s)
I20260926 06:59:45.883605 15648 tablet_bootstrap.cc:492] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:59:45.883951 15648 tablet_bootstrap.cc:492] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Bootstrap complete.
I20260926 06:59:45.885160 15648 ts_tablet_manager.cc:1397] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.010s
I20260926 06:59:45.886619 15648 raft_consensus.cc:348] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.886956 15648 raft_consensus.cc:729] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444edb1aa920424395882c1f96d49299, State: Initialized, Role: FOLLOWER
I20260926 06:59:45.887111 15648 consensus_queue.cc:260] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:45.887403 15648 ts_tablet_manager.cc:1428] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:45.895800 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.881254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:41541)
0926 06:59:45.881780 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.881830 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.882504 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.883296 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.883300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.883767 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:45.884099 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.884105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.885990 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.885991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.887592 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.887595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.888421 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.888422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.888577 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.889229 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:45.889239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:45.891054 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:45.893658 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.893665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.893668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.893970 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.894401 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.894402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.894403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.894549 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.895031 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:45.895036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:45.895211 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.895364 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.895561 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":110,"threads_started":1}
I20260926 06:59:45.895815 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.881563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:41541 (local address 127.1.74.126:42169)
0926 06:59:45.882724 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.882734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.882773 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:59:45.882910 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.882916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.882974 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.883108 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:45.884257 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.885833 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.887695 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.888309 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.888425 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.891179 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:45.891183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:45.891183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:45.891198 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:45.893551 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.894063 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.894066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.894066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.894197 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.894628 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.894630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.894630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.894946 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:45.895035 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.895498 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.895636 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":82,"threads_started":1}
I20260926 06:59:45.897192 15756 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42169
I20260926 06:59:45.897300 15756 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:45.897523 15756 heartbeater.cc:511] Master 127.1.74.126:42169 requested a full tablet report, sending...
I20260926 06:59:45.898276 15568 ts_manager.cc:194] Registered new tserver with Master: 444edb1aa920424395882c1f96d49299 (127.1.74.65:44825)
I20260926 06:59:45.899441 15568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:41541
I20260926 06:59:45.900400 15756 heartbeater.cc:503] Master 127.1.74.126:42169 was elected leader, sending a full tablet report...
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 06:59:45.908680 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.899203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.1.74.65:44825)
0926 06:59:45.899934 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.899938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.899949 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:59:45.899989 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.899992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.900008 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.900100 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:45.900924 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.901979 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.903136 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.903596 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.903665 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.905950 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:45.905957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:45.905957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:45.905963 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:45.907472 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.907951 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.907953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.907954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.908004 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:45.908195 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:45.908195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:45.908196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:45.908306 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:45.908363 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:45.908545 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 06:59:45.908606 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":106,"threads_started":1}
I20260926 06:59:45.908738 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.898895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44825 (local address 127.0.0.1:58466)
0926 06:59:45.899406 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.899445 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.899531 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.900156 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.900161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.900690 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:45.900836 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.900838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.902082 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.902083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.903037 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.903040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.903666 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.903668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:45.903741 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:45.904262 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:45.904267 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:45.905838 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:45.907551 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.907559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.907561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.907842 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.908066 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:45.908067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:45.908068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:45.908135 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:45.908367 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:45.908373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:45.908427 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 06:59:45.908523 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 06:59:45.908645 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":127,"threads_started":1}
I20260926 06:59:45.909196 15684 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4e3ff7974e4eaab1df2b250d6cb20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:45 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:58466
I20260926 06:59:45.909868 15765 tablet_replica.cc:330] stopping tablet replica
I20260926 06:59:45.909969 15765 raft_consensus.cc:2229] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:59:45.910037 15765 raft_consensus.cc:2258] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:59:45.910321 15765 ts_tablet_manager.cc:1905] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:59:45.912633 15765 ts_tablet_manager.cc:1918] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:59:45.912691 15765 log.cc:1196] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/ba4e3ff7974e4eaab1df2b250d6cb20d
I20260926 06:59:45.912930 15765 ts_tablet_manager.cc:1939] T ba4e3ff7974e4eaab1df2b250d6cb20d P 444edb1aa920424395882c1f96d49299: Deleting consensus metadata
I20260926 06:59:45.913374 15554 catalog_manager.cc:4841] TS 444edb1aa920424395882c1f96d49299 (127.1.74.65:44825): tablet ba4e3ff7974e4eaab1df2b250d6cb20d (table test-table [id=e6e0d9a0778047ca881e2385190c07f2]) successfully deleted
W20260926 06:59:45.961764 15758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:45.961992 15758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:45.962028 15758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:45.963174 15758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:45.963246 15758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:45.963315 15758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:59:45.964761 15758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:40191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=41309
--tserver_master_addrs=127.1.74.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:45.965018 15758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:45.965248 15758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:45.970135 15771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:45.970139 15770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.970216 15758 server_base.cc:1034] running on GCE node
W20260926 06:59:45.970144 15773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:45.970584 15758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:45.970829 15758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:45.971987 15758 hybrid_clock.cc:648] HybridClock initialized: now 1790405985971981 us; error 36 us; skew 500 ppm
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:45.974309 15758 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:59:45.974713 15758 webserver.cc:460] Webserver started at http://127.1.74.66:41309/ using document root <none> and password file <none>
I20260926 06:59:45.974932 15758 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:45.975010 15758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:45.976399 15758 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:45.976958 15780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.977123 15758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:59:45.977214 15758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "2182d3ae26574f9caedcdf253178c5cd"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "88d05f99dbe546331da9771cd83bd25f"
server_key_iv: "3523cfc1290a6d89d96055b96759b7d2"
server_key_version: "encryptionkey@0"
I20260926 06:59:45.977558 15758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:45.988499 15758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:45.990161 15758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:45.990311 15758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:45.990507 15758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:45.990782 15758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:45.990847 15758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.990885 15758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:45.990911 15758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:45.996544 15758 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:40191
I20260926 06:59:45.996603 15893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:40191 every 8 connection(s)
I20260926 06:59:45.996929 15758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:59:45 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405985, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:46.005658 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.997643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:56983 (local address 127.1.74.126:42169)
0926 06:59:45.997685 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:45.997687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:45.997885 (+ 198us) server_negotiation.cc:407] Connection header received
0926 06:59:45.998322 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:45.998322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:45.998332 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:45.998392 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:45.999319 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.999820 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.000728 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.000916 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.000965 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.002936 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:46.002938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:46.002938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:46.002945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:46.004550 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:46.004915 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:46.004916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:46.004917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:46.004965 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:46.005166 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:46.005166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:46.005166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:46.005266 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:46.005314 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:46.005529 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 06:59:46.005593 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:59:46.005810 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:45.997603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:56983)
0926 06:59:45.997749 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:45.997797 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:45.998244 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:45.998442 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:45.998445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:45.999029 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:45.999204 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:45.999209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:45.999900 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:45.999900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.000658 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.000659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.000986 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.000987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.001082 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.001510 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:46.001517 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:46.002847 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:46.004619 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:46.004625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:46.004626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:46.004844 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:46.005023 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:46.005023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:46.005024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:46.005113 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:46.005317 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:46.005321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:46.005426 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 06:59:46.005502 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 06:59:46.005616 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260926 06:59:46.006259 15894 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42169
I20260926 06:59:46.006348 15894 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:46.006524 15894 heartbeater.cc:511] Master 127.1.74.126:42169 requested a full tablet report, sending...
I20260926 06:59:46.006817 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 15758
I20260926 06:59:46.006881 15568 ts_manager.cc:194] Registered new tserver with Master: 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191)
I20260926 06:59:46.007083 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:45523
--local_ip_for_outbound_sockets=127.1.74.67
--tserver_master_addrs=127.1.74.126:42169
--webserver_port=46739
--webserver_interface=127.1.74.67
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:46.007359 15568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:56983
W20260926 06:59:46.088928 15897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:46.089125 15897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:46.089148 15897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:46.090337 15897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:46.090387 15897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:46.090442 15897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:59:46.091989 15897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:45523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=46739
--tserver_master_addrs=127.1.74.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:46.092229 15897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:46.092461 15897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:46.097386 15903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:46.097508 15897 server_base.cc:1034] running on GCE node
W20260926 06:59:46.097431 15902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:46.097429 15905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:46.097877 15897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:46.098140 15897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:46.099319 15897 hybrid_clock.cc:648] HybridClock initialized: now 1790405986099302 us; error 40 us; skew 500 ppm
Sep 26 06:59:46 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:46.101876 15897 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:59:46.102326 15897 webserver.cc:460] Webserver started at http://127.1.74.67:46739/ using document root <none> and password file <none>
I20260926 06:59:46.102527 15897 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:46.102581 15897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:46.104038 15897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:59:46.104612 15912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.104799 15897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:46.104925 15897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "a5027532a86d4155b0c79e89dc7bde51"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "52bd42f294ebac4da136d6b81243fb75"
server_key_iv: "4923a82b156b18bcb19af09bd3cfec7d"
server_key_version: "encryptionkey@0"
I20260926 06:59:46.105259 15897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:46.136085 15897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:46.138680 15897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:46.138873 15897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:46.139127 15897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:46.139525 15897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:46.139591 15897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.139633 15897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:46.139659 15897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.145648 15897 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:45523
I20260926 06:59:46.145694 16025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:45523 every 8 connection(s)
I20260926 06:59:46.146034 15897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:59:46 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 06:59:46.152292 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 15897
I20260926 06:59:46.155184 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.146750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:42163 (local address 127.1.74.126:42169)
0926 06:59:46.146819 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:46.146822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:46.147042 (+ 220us) server_negotiation.cc:407] Connection header received
0926 06:59:46.147587 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:46.147588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:46.147598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:46.147651 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:46.148600 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.149073 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.150061 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.150243 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.150292 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.152273 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:46.152275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:46.152275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:46.152283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:46.153728 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:46.154186 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:46.154187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:46.154187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:46.154236 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:46.154544 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:46.154545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:46.154545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:46.154665 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:46.154740 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:46.155008 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 06:59:46.155093 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:59:46.155403 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.146739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:42163)
0926 06:59:46.146921 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.146957 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:46.147499 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:46.147705 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:46.147708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:46.148326 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:46.148528 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.148533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.149148 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.149148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.149990 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.149991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.150310 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.150311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.150405 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.150837 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:46.150844 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:46.152178 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:46.153802 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:46.153809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:46.153811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:46.154112 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:46.154297 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:46.154298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:46.154298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:46.154431 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:46.154731 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:46.154737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:46.154869 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 06:59:46.154987 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 06:59:46.155146 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260926 06:59:46.155917 16026 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42169
I20260926 06:59:46.156023 16026 heartbeater.cc:464] Registering TS with master...
I20260926 06:59:46.156208 16026 heartbeater.cc:511] Master 127.1.74.126:42169 requested a full tablet report, sending...
I20260926 06:59:46.156611 15568 ts_manager.cc:194] Registered new tserver with Master: a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523)
I20260926 06:59:46.157104 15568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:42163
I20260926 06:59:46.159560 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.152676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.1.74.126:42169)
0926 06:59:46.152970 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:46.152971 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:46.152977 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:59:46.152994 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:46.152995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:46.153003 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:46.153081 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:46.153631 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.154395 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.155138 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.155498 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.155562 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.157048 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:46.157049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:46.157049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:46.157056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:46.158453 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:46.158825 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:46.158826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:46.158827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:46.158908 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:46.159086 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:46.159087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:46.159087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:46.159173 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:46.159354 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:46.159385 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 06:59:46.159448 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":66,"threads_started":1}
I20260926 06:59:46.160233 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:59:46.162802 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.160634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.1.74.126:42169)
0926 06:59:46.160697 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:46.160699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:46.160758 (+ 59us) server_negotiation.cc:407] Connection header received
0926 06:59:46.160802 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:46.160802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:46.160830 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:46.160886 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:59:46.161001 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.161447 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.162153 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.162329 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.162377 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.162508 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:59:46.162600 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:59:46.162659 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:46.162690 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 06:59:46.162709 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:59:46.163115 15568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:59:46.164824 15568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:59:46.167212 15684 tablet_service.cc:1467] Processing CreateTablet for tablet 91f2c34048e9448fab46835d5c9fe0b3 (DEFAULT_TABLE table=test-table [id=f91cb4468a69432cb622be7cbf987cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:59:46.167479 15684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f2c34048e9448fab46835d5c9fe0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:46.168919 15648 tablet_bootstrap.cc:492] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Bootstrap starting.
I20260926 06:59:46.169346 15648 tablet_bootstrap.cc:654] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:46.170281 15648 tablet_bootstrap.cc:492] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: No bootstrap required, opened a new log
I20260926 06:59:46.170346 15648 ts_tablet_manager.cc:1397] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:59:46.170508 15648 raft_consensus.cc:348] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.170569 15648 raft_consensus.cc:374] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:46.170599 15648 raft_consensus.cc:729] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444edb1aa920424395882c1f96d49299, State: Initialized, Role: FOLLOWER
I20260926 06:59:46.170658 15648 consensus_queue.cc:260] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.170751 15648 ts_tablet_manager.cc:1428] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.171453 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.166732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:45523 (local address 127.0.0.1:39960)
0926 06:59:46.166792 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.166803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:46.166842 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:46.167418 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:46.167419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:46.167420 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:46.167462 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.167463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.168724 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.168725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.169997 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.170000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.171164 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.171165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.171244 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.171291 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:59:46.171325 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 06:59:46.171370 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":27}
I20260926 06:59:46.171766 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.166833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.1.74.67:45523)
0926 06:59:46.167216 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:46.167220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:46.167229 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:59:46.167290 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:46.167292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:46.167307 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:46.167540 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:46.167607 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.168637 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.170097 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.171081 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.171547 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.171619 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:46.171647 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 06:59:46.171688 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":91,"threads_started":1}
I20260926 06:59:46.172294 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.166732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:40191 (local address 127.0.0.1:48806)
0926 06:59:46.166922 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.166933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:46.166986 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:46.167342 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:46.167343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:46.167346 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:46.167393 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.167394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.168696 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.168697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.171033 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.171036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.172060 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.172060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.172128 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.172132 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:59:46.172182 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:59:46.172223 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1272,"thread_start_us":104,"threads_started":1}
I20260926 06:59:46.172345 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.166947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.1.74.66:40191)
0926 06:59:46.167129 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:46.167135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:46.167154 (+ 19us) server_negotiation.cc:407] Connection header received
0926 06:59:46.167198 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:46.167201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:46.167217 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:46.167330 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:46.167458 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.168577 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.171132 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.171887 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.171968 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.172027 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:46.172218 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 06:59:46.172265 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260926 06:59:46.172544 15960 tablet_service.cc:1467] Processing CreateTablet for tablet 91f2c34048e9448fab46835d5c9fe0b3 (DEFAULT_TABLE table=test-table [id=f91cb4468a69432cb622be7cbf987cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:59:46.172879 15960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f2c34048e9448fab46835d5c9fe0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:46.173018 15828 tablet_service.cc:1467] Processing CreateTablet for tablet 91f2c34048e9448fab46835d5c9fe0b3 (DEFAULT_TABLE table=test-table [id=f91cb4468a69432cb622be7cbf987cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:59:46.173303 15828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91f2c34048e9448fab46835d5c9fe0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:59:46.175953 16036 tablet_bootstrap.cc:492] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: Bootstrap starting.
I20260926 06:59:46.176180 16037 tablet_bootstrap.cc:492] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: Bootstrap starting.
I20260926 06:59:46.176589 16036 tablet_bootstrap.cc:654] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:46.176769 16037 tablet_bootstrap.cc:654] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: Neither blocks nor log segments found. Creating new log.
I20260926 06:59:46.177150 16036 log.cc:824] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:46.177254 16037 log.cc:824] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:46.177850 16036 tablet_bootstrap.cc:492] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: No bootstrap required, opened a new log
I20260926 06:59:46.177909 16036 ts_tablet_manager.cc:1397] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:59:46.177975 16037 tablet_bootstrap.cc:492] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: No bootstrap required, opened a new log
I20260926 06:59:46.178037 16037 ts_tablet_manager.cc:1397] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:59:46.179610 16036 raft_consensus.cc:348] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.179747 16036 raft_consensus.cc:374] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:46.179770 16036 raft_consensus.cc:729] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5027532a86d4155b0c79e89dc7bde51, State: Initialized, Role: FOLLOWER
I20260926 06:59:46.179647 16037 raft_consensus.cc:348] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.179811 16037 raft_consensus.cc:374] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:59:46.179853 16037 raft_consensus.cc:729] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2182d3ae26574f9caedcdf253178c5cd, State: Initialized, Role: FOLLOWER
I20260926 06:59:46.179920 16036 consensus_queue.cc:260] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.179973 16037 consensus_queue.cc:260] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.180238 16036 ts_tablet_manager.cc:1428] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:59:46.180326 15894 heartbeater.cc:503] Master 127.1.74.126:42169 was elected leader, sending a full tablet report...
I20260926 06:59:46.180238 16026 heartbeater.cc:503] Master 127.1.74.126:42169 was elected leader, sending a full tablet report...
I20260926 06:59:46.180621 16037 ts_tablet_manager.cc:1428] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:59:46.215476 16041 raft_consensus.cc:482] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:59:46.215642 16041 raft_consensus.cc:504] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.215958 16041 leader_election.cc:288] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523), 444edb1aa920424395882c1f96d49299 (127.1.74.65:44825)
I20260926 06:59:46.218873 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.216265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:48791 (local address 127.1.74.67:45523)
0926 06:59:46.216341 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:46.216343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:46.216470 (+ 127us) server_negotiation.cc:407] Connection header received
0926 06:59:46.216496 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:46.216497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:46.216509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:46.216571 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:46.216753 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.217197 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.218171 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.218530 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.218583 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.218591 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:46.218753 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 06:59:46.218789 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:59:46.218899 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.216149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:45523 (local address 127.1.74.66:48791)
0926 06:59:46.216381 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.216390 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:46.216451 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:46.216581 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:46.216581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:46.216582 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:46.216678 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.216680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.217298 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.217299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.218055 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.218057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.218609 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.218611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.218683 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.218687 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:59:46.218739 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 06:59:46.218783 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":59,"thread_start_us":120,"threads_started":1}
I20260926 06:59:46.218979 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.216250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:37789 (local address 127.1.74.65:44825)
0926 06:59:46.216306 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:46.216308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:46.216317 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:59:46.216349 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:46.216350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:46.216378 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:46.216455 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:59:46.216657 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.217191 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.218134 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.218590 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.218660 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.218746 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:46.218858 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 06:59:46.218912 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 06:59:46.219169 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.216149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44825 (local address 127.1.74.66:37789)
0926 06:59:46.216214 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.216228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:46.216280 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:46.216485 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:46.216486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:46.216488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:59:46.216569 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.216571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.217280 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.217281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.218036 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:46.218037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.218684 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.218685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:46.218777 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.218781 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:59:46.218845 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:59:46.218885 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 06:59:46.219307 15980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f2c34048e9448fab46835d5c9fe0b3" candidate_uuid: "2182d3ae26574f9caedcdf253178c5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5027532a86d4155b0c79e89dc7bde51" is_pre_election: true
I20260926 06:59:46.219470 15980 raft_consensus.cc:2454] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2182d3ae26574f9caedcdf253178c5cd in term 0.
I20260926 06:59:46.219525 15710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f2c34048e9448fab46835d5c9fe0b3" candidate_uuid: "2182d3ae26574f9caedcdf253178c5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444edb1aa920424395882c1f96d49299" is_pre_election: true
I20260926 06:59:46.219693 15710 raft_consensus.cc:2454] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2182d3ae26574f9caedcdf253178c5cd in term 0.
I20260926 06:59:46.219715 15781 leader_election.cc:302] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2182d3ae26574f9caedcdf253178c5cd, a5027532a86d4155b0c79e89dc7bde51; no voters:
I20260926 06:59:46.219857 16041 raft_consensus.cc:2781] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:59:46.219950 16041 raft_consensus.cc:482] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:59:46.219985 16041 raft_consensus.cc:3037] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:46.220672 16041 raft_consensus.cc:504] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.220825 16041 leader_election.cc:288] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [CANDIDATE]: Term 1 election: Requested vote from peers a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523), 444edb1aa920424395882c1f96d49299 (127.1.74.65:44825)
I20260926 06:59:46.220994 15980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f2c34048e9448fab46835d5c9fe0b3" candidate_uuid: "2182d3ae26574f9caedcdf253178c5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5027532a86d4155b0c79e89dc7bde51"
I20260926 06:59:46.221064 15710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91f2c34048e9448fab46835d5c9fe0b3" candidate_uuid: "2182d3ae26574f9caedcdf253178c5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444edb1aa920424395882c1f96d49299"
I20260926 06:59:46.221093 15980 raft_consensus.cc:3037] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:46.221148 15710 raft_consensus.cc:3037] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:59:46.221977 15980 raft_consensus.cc:2454] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2182d3ae26574f9caedcdf253178c5cd in term 1.
I20260926 06:59:46.221977 15710 raft_consensus.cc:2454] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2182d3ae26574f9caedcdf253178c5cd in term 1.
I20260926 06:59:46.222143 15781 leader_election.cc:302] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2182d3ae26574f9caedcdf253178c5cd, a5027532a86d4155b0c79e89dc7bde51; no voters:
I20260926 06:59:46.222239 16041 raft_consensus.cc:2781] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:59:46.222396 16041 raft_consensus.cc:686] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 1 LEADER]: Becoming Leader. State: Replica: 2182d3ae26574f9caedcdf253178c5cd, State: Running, Role: LEADER
I20260926 06:59:46.222492 16041 consensus_queue.cc:237] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.223392 15567 catalog_manager.cc:5495] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66). New cstate: current_term: 1 leader_uuid: "2182d3ae26574f9caedcdf253178c5cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } health_report { overall_health: UNKNOWN } } }
I20260926 06:59:46.228363 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.226265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.1.74.66:40191)
0926 06:59:46.226335 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:46.226337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:46.226355 (+ 18us) server_negotiation.cc:407] Connection header received
0926 06:59:46.226380 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:46.226381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:46.226396 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:46.226447 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:59:46.226558 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.226988 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.227716 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:46.227893 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:46.227944 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:46.228063 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:59:46.228148 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:59:46.228206 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:46.228237 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 06:59:46.228252 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:59:46.229950 15980 raft_consensus.cc:1261] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [term 1 FOLLOWER]: Refusing update from remote peer 2182d3ae26574f9caedcdf253178c5cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:59:46.229951 15710 raft_consensus.cc:1261] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Refusing update from remote peer 2182d3ae26574f9caedcdf253178c5cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:59:46.230283 16043 consensus_queue.cc:1035] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:59:46.230388 16041 consensus_queue.cc:1035] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:59:46.232544 16047 mvcc.cc:205] Tried to move back new op lower bound from 7333502919595409408 to 7333502919568044032. Current Snapshot: MvccSnapshot[applied={T|T < 7333502919595409408}]
I20260926 06:59:46.232800 16048 mvcc.cc:205] Tried to move back new op lower bound from 7333502919595409408 to 7333502919568044032. Current Snapshot: MvccSnapshot[applied={T|T < 7333502919595409408}]
I20260926 06:59:46.234313 15567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:59:46.234409 15567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:59:46.235136 15567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Sending DeleteTablet for 3 replicas of tablet 91f2c34048e9448fab46835d5c9fe0b3
I20260926 06:59:46.235486 15828 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f2c34048e9448fab46835d5c9fe0b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:46 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:48806
I20260926 06:59:46.235512 15960 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f2c34048e9448fab46835d5c9fe0b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:46 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:39960
I20260926 06:59:46.235672 15684 tablet_service.cc:1514] Processing DeleteTablet for tablet 91f2c34048e9448fab46835d5c9fe0b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:59:46 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:58466
I20260926 06:59:46.235756 15765 tablet_replica.cc:330] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: stopping tablet replica
I20260926 06:59:46.235770 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 15606
I20260926 06:59:46.235803 15765 raft_consensus.cc:2229] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:59:46.235791 16055 tablet_replica.cc:330] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: stopping tablet replica
I20260926 06:59:46.235850 15765 raft_consensus.cc:2258] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:59:46.235884 16055 raft_consensus.cc:2229] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:59:46.235989 16055 raft_consensus.cc:2258] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:59:46.236109 16056 tablet_replica.cc:330] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: stopping tablet replica
I20260926 06:59:46.236177 16056 raft_consensus.cc:2229] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:59:46.236513 16056 raft_consensus.cc:2258] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:59:46.236943 16056 ts_tablet_manager.cc:1905] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:59:46.238629 16055 ts_tablet_manager.cc:1905] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:59:46.238651 16056 ts_tablet_manager.cc:1918] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:59:46.238721 16056 log.cc:1196] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/91f2c34048e9448fab46835d5c9fe0b3
I20260926 06:59:46.239109 16056 ts_tablet_manager.cc:1939] T 91f2c34048e9448fab46835d5c9fe0b3 P a5027532a86d4155b0c79e89dc7bde51: Deleting consensus metadata
I20260926 06:59:46.239638 15552 catalog_manager.cc:4841] TS a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523): tablet 91f2c34048e9448fab46835d5c9fe0b3 (table test-table [id=f91cb4468a69432cb622be7cbf987cbc]) successfully deleted
I20260926 06:59:46.240527 16055 ts_tablet_manager.cc:1918] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:59:46.240608 16055 log.cc:1196] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/91f2c34048e9448fab46835d5c9fe0b3
I20260926 06:59:46.240913 16055 ts_tablet_manager.cc:1939] T 91f2c34048e9448fab46835d5c9fe0b3 P 2182d3ae26574f9caedcdf253178c5cd: Deleting consensus metadata
I20260926 06:59:46.241284 15554 catalog_manager.cc:4841] TS 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191): tablet 91f2c34048e9448fab46835d5c9fe0b3 (table test-table [id=f91cb4468a69432cb622be7cbf987cbc]) successfully deleted
W20260926 06:59:46.242599 15554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:44825 (error 108)
I20260926 06:59:46.242868 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 15758
I20260926 06:59:46.243021 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.242898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44825 (local address 127.0.0.1:58472)
0926 06:59:46.242955 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.242992 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 06:59:46.243752 15554 catalog_manager.cc:4584] TS 444edb1aa920424395882c1f96d49299 (127.1.74.65:44825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91f2c34048e9448fab46835d5c9fe0b3: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44825: connect: Connection refused (error 111)
I20260926 06:59:46.249162 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 15897
I20260926 06:59:46.253476 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 15535
I20260926 06:59:46.258564 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42169
--webserver_interface=127.1.74.126
--webserver_port=33061
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:42169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:59:46.339586 16058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:46.339790 16058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:46.339812 16058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:46.340904 16058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:59:46.340941 16058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:46.340955 16058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:46.340968 16058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:59:46.340981 16058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:59:46.342371 16058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:42169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=33061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:46.342561 16058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:46.342725 16058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:46.347456 16063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:46.347461 16066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:46.347468 16064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:46.347617 16058 server_base.cc:1034] running on GCE node
I20260926 06:59:46.348007 16058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:46.348294 16058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:46.349457 16058 hybrid_clock.cc:648] HybridClock initialized: now 1790405986349449 us; error 42 us; skew 500 ppm
Sep 26 06:59:46 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:46.351897 16058 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 06:59:46.352278 16058 webserver.cc:460] Webserver started at http://127.1.74.126:33061/ using document root <none> and password file <none>
I20260926 06:59:46.352464 16058 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:46.352505 16058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:46.353933 16058 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:46.354566 16073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.354770 16058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:46.354859 16058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "ecd8cbf76b404434af5fbe95c6fc83c3"
format_stamp: "Formatted at 2026-09-26 06:59:44 on dist-test-slave-sfvt"
server_key: "002b10368d5af72c3368f4692891ecf5"
server_key_iv: "bd5587393f779016adead3bc299dee20"
server_key_version: "encryptionkey@0"
I20260926 06:59:46.355195 16058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:46.381556 16058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:46.383555 16058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:46.383671 16058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:46.387450 16058 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:42169
I20260926 06:59:46.387514 16125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:42169 every 8 connection(s)
I20260926 06:59:46.387888 16058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 06:59:46.389467 16126 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:59:46.390137 16126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Bootstrap starting.
I20260926 06:59:46.393685 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 16058
I20260926 06:59:46.393962 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:44825
--local_ip_for_outbound_sockets=127.1.74.65
--tserver_master_addrs=127.1.74.126:42169
--webserver_port=46185
--webserver_interface=127.1.74.65
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:46.398950 16126 log.cc:824] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:46.401234 16126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:59:46.401552 16126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3: Bootstrap complete.
I20260926 06:59:46.403872 16126 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:46.404125 16126 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd8cbf76b404434af5fbe95c6fc83c3, State: Initialized, Role: FOLLOWER
I20260926 06:59:46.404327 16126 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:46.404424 16126 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:59:46.404479 16126 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:59:46.404525 16126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 2 FOLLOWER]: Advancing to term 3
I20260926 06:59:46.405507 16126 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:46.405627 16126 leader_election.cc:302] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecd8cbf76b404434af5fbe95c6fc83c3; no voters:
I20260926 06:59:46.405838 16126 leader_election.cc:288] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 06:59:46.405872 16130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 06:59:46.405982 16130 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [term 3 LEADER]: Becoming Leader. State: Replica: ecd8cbf76b404434af5fbe95c6fc83c3, State: Running, Role: LEADER
I20260926 06:59:46.406117 16126 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:59:46.406136 16130 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } }
I20260926 06:59:46.406591 16131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } } }
I20260926 06:59:46.406615 16132 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecd8cbf76b404434af5fbe95c6fc83c3. Latest consensus state: current_term: 3 leader_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd8cbf76b404434af5fbe95c6fc83c3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42169 } } }
I20260926 06:59:46.406716 16132 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:46.406674 16131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd8cbf76b404434af5fbe95c6fc83c3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:59:46.406971 16136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:59:46.407910 16136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:59:46.408066 16136 catalog_manager.cc:1259] Loaded cluster ID: cc792b77d7fb41d3915cab7f1f29de0c
I20260926 06:59:46.408311 16136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 06:59:46.408596 16136 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f85a225b9ad google::LogMessage::Fail() at ??:0
@ 0x7f85a225e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f85a225b4de google::LogMessage::Flush() at ??:0
@ 0x7f85a225ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f85aa17c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f85aa1858b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f85a41290d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f85a411e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f85a4f826db start_thread at ??:0
@ 0x7f85a0d5961f clone at ??:0
W20260926 06:59:46.483801 16128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:46.484026 16128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:46.484066 16128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:46.485205 16128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:46.485276 16128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:46.485322 16128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 06:59:46.486763 16128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:44825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=46185
--tserver_master_addrs=127.1.74.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:46.487020 16128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:46.487249 16128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:46.492178 16165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:46.492165 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:46.492296 16128 server_base.cc:1034] running on GCE node
W20260926 06:59:46.492162 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:46.492657 16128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:46.492931 16128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:46.494103 16128 hybrid_clock.cc:648] HybridClock initialized: now 1790405986494092 us; error 30 us; skew 500 ppm
Sep 26 06:59:46 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:46.496671 16128 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 06:59:46.497097 16128 webserver.cc:460] Webserver started at http://127.1.74.65:46185/ using document root <none> and password file <none>
I20260926 06:59:46.497339 16128 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:46.497408 16128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:46.498713 16128 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:59:46.499492 16172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.499723 16128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:59:46.499825 16128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "444edb1aa920424395882c1f96d49299"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "af48b5ad833413bfcc150a73c1748f58"
server_key_iv: "2b1a41a86c60158059c560c23c79254c"
server_key_version: "encryptionkey@0"
I20260926 06:59:46.500200 16128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:46.526636 16128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:46.528957 16128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:46.529134 16128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:46.529424 16128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:46.529881 16179 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:59:46.531952 16128 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:59:46.532018 16128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:59:46.532063 16128 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:59:46.532717 16128 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:59:46.532775 16128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:59:46.532855 16179 tablet_bootstrap.cc:492] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Bootstrap starting.
I20260926 06:59:46.540215 16128 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:44825
I20260926 06:59:46.540355 16286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:44825 every 8 connection(s)
I20260926 06:59:46.540654 16128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 06:59:46.543942 16179 log.cc:824] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Log is configured to *not* fsync() on all Append() calls
I20260926 06:59:46.546636 16179 tablet_bootstrap.cc:492] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:59:46.546948 16179 tablet_bootstrap.cc:492] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Bootstrap complete.
I20260926 06:59:46.548240 16179 ts_tablet_manager.cc:1397] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.010s
I20260926 06:59:46.549605 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 16128
I20260926 06:59:46.549862 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:40191
--local_ip_for_outbound_sockets=127.1.74.66
--tserver_master_addrs=127.1.74.126:42169
--webserver_port=41309
--webserver_interface=127.1.74.66
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:46.549669 16179 raft_consensus.cc:348] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.549983 16179 raft_consensus.cc:729] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444edb1aa920424395882c1f96d49299, State: Initialized, Role: FOLLOWER
I20260926 06:59:46.550145 16179 consensus_queue.cc:260] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:46.550427 16179 ts_tablet_manager.cc:1428] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260926 06:59:46.613070 16289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:59:46.541323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:36717)
0926 06:59:46.541500 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.541545 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:46.542026 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:46.612815 (+ 70789us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
W20260926 06:59:46.613385 16287 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:42169 (0 consecutive failures): Network error: Failed to ping master at 127.1.74.126:42169: Client connection negotiation failed: client connection to 127.1.74.126:42169: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:59:46.613776 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.613645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:58035)
0926 06:59:46.613700 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.613748 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 06:59:46.614161 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.614053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:53951)
0926 06:59:46.614099 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.614132 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 06:59:46.614307 16287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:59:46.631795 16292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:46.631994 16292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:46.632017 16292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:46.633155 16292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:46.633196 16292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:46.633222 16292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 06:59:46.634640 16292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:40191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=41309
--tserver_master_addrs=127.1.74.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:46.634877 16292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:46.635093 16292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:46.640108 16297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:46.640100 16298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:46.640125 16300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:46.640354 16292 server_base.cc:1034] running on GCE node
I20260926 06:59:46.640561 16292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:46.640746 16292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:46.641913 16292 hybrid_clock.cc:648] HybridClock initialized: now 1790405986641897 us; error 31 us; skew 500 ppm
Sep 26 06:59:46 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:46.644285 16292 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 06:59:46.644759 16292 webserver.cc:460] Webserver started at http://127.1.74.66:41309/ using document root <none> and password file <none>
I20260926 06:59:46.645015 16292 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:46.645080 16292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:46.646348 16292 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:59:46.646893 16307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.647068 16292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:59:46.647159 16292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "2182d3ae26574f9caedcdf253178c5cd"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "88d05f99dbe546331da9771cd83bd25f"
server_key_iv: "3523cfc1290a6d89d96055b96759b7d2"
server_key_version: "encryptionkey@0"
I20260926 06:59:46.647565 16292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:46.664719 16292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:46.666509 16292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:46.666661 16292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:46.666878 16292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:46.667171 16292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:46.667220 16292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.667279 16292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:46.667308 16292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.673193 16292 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:40191
I20260926 06:59:46.673240 16420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:40191 every 8 connection(s)
I20260926 06:59:46.673559 16292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 06:59:46.674511 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.674114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:36187)
0926 06:59:46.674253 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.674362 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260926 06:59:46.674903 16421 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:42169 (0 consecutive failures): Network error: Failed to ping master at 127.1.74.126:42169: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
I20260926 06:59:46.675232 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.675127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:46699)
0926 06:59:46.675175 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.675207 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 06:59:46.675231 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 16292
I20260926 06:59:46.675529 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:45523
--local_ip_for_outbound_sockets=127.1.74.67
--tserver_master_addrs=127.1.74.126:42169
--webserver_port=46739
--webserver_interface=127.1.74.67
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:59:46.675633 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.675524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:45683)
0926 06:59:46.675575 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.675605 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 06:59:46.675724 16421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:59:46.755767 16424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:59:46.755996 16424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:59:46.756038 16424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:59:46.757167 16424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:59:46.757232 16424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:59:46.757275 16424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 06:59:46.758734 16424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:45523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=46739
--tserver_master_addrs=127.1.74.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 06:59:46.758994 16424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:59:46.759217 16424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:59:46.764809 16429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:46.764822 16430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:59:46.764853 16432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:59:46.764853 16424 server_base.cc:1034] running on GCE node
I20260926 06:59:46.765251 16424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:59:46.765493 16424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:59:46.766636 16424 hybrid_clock.cc:648] HybridClock initialized: now 1790405986766632 us; error 35 us; skew 500 ppm
Sep 26 06:59:46 dist-test-slave-sfvt krb5kdc[14987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:59:46.769156 16424 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 06:59:46.769625 16424 webserver.cc:460] Webserver started at http://127.1.74.67:46739/ using document root <none> and password file <none>
I20260926 06:59:46.769855 16424 fs_manager.cc:359] Metadata directory not provided
I20260926 06:59:46.769899 16424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:59:46.771229 16424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:59:46.771908 16439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.772092 16424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:59:46.772179 16424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "a5027532a86d4155b0c79e89dc7bde51"
format_stamp: "Formatted at 2026-09-26 06:59:45 on dist-test-slave-sfvt"
server_key: "52bd42f294ebac4da136d6b81243fb75"
server_key_iv: "4923a82b156b18bcb19af09bd3cfec7d"
server_key_version: "encryptionkey@0"
I20260926 06:59:46.772516 16424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:59:46.786387 16424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:59:46.788240 16424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:59:46.788395 16424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:59:46.788611 16424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:59:46.788918 16424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:59:46.788980 16424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.789021 16424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:59:46.789049 16424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:59:46.794766 16424 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:45523
I20260926 06:59:46.794814 16552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:45523 every 8 connection(s)
I20260926 06:59:46.795136 16424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 06:59:46.796222 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.795782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:47243)
0926 06:59:46.795929 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.796018 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
W20260926 06:59:46.796662 16553 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:42169 (0 consecutive failures): Network error: Failed to ping master at 127.1.74.126:42169: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
I20260926 06:59:46.796970 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.796861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:37687)
0926 06:59:46.796898 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.796930 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 06:59:46.797245 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:46.797160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:44929)
0926 06:59:46.797205 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:46.797227 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 06:59:46.797349 16553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:59:46.799937 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 16424
I20260926 06:59:47.614929 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:47.614702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:34013)
0926 06:59:47.614833 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:47.614869 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260926 06:59:47.676234 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:47.676021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:38327)
0926 06:59:47.676149 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:47.676183 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260926 06:59:47.797825 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:47.797626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:49099)
0926 06:59:47.797746 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:47.797777 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":52,"threads_started":1}
I20260926 06:59:48.356572 16560 raft_consensus.cc:482] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:59:48.356691 16560 raft_consensus.cc:504] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:48.357055 16560 leader_election.cc:288] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523), 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191)
Sep 26 06:59:48 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.67@KRBTEST.COM
Sep 26 06:59:48 dist-test-slave-sfvt krb5kdc[14987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790405986, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.66@KRBTEST.COM
I20260926 06:59:48.369127 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:48.357343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:53045 (local address 127.1.74.66:40191)
0926 06:59:48.357520 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:48.357534 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:48.357590 (+ 56us) server_negotiation.cc:407] Connection header received
0926 06:59:48.358191 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:48.358193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:48.358246 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:48.358403 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:48.359402 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:48.360780 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:48.362292 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:48.362832 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:48.362938 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:48.365713 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:48.365715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:48.365716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:48.365724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:48.367509 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:48.367907 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:48.367910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:48.367911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:48.368002 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:48.368193 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:48.368194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:48.368194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:48.368446 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:48.368533 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:48.368883 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 06:59:48.369017 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
I20260926 06:59:48.369149 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:48.357391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:54081 (local address 127.1.74.67:45523)
0926 06:59:48.357542 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 06:59:48.357549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:59:48.357590 (+ 41us) server_negotiation.cc:407] Connection header received
0926 06:59:48.358181 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:59:48.358186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:59:48.358246 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:59:48.358404 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:59:48.359402 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:48.360784 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:48.362326 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:59:48.362833 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:59:48.362951 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:48.365308 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:59:48.365312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:59:48.365312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:59:48.365322 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:59:48.367308 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:48.367766 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:48.367768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:48.367769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:48.367849 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:59:48.368123 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:59:48.368124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:59:48.368124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:59:48.368446 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:59:48.368533 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:59:48.368883 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 06:59:48.369017 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260926 06:59:48.369148 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:48.357228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:45523 (local address 127.1.74.65:54081)
0926 06:59:48.357504 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:48.357517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:48.357601 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:48.358466 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:48.358469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:48.358989 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:48.359295 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:48.359296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:48.360902 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:48.360902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:48.362205 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:48.362207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:48.363024 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:48.363025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:48.363081 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:48.363670 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:48.363678 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:48.365204 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:48.367384 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:48.367390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:48.367392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:48.367687 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:48.367939 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:48.367939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:48.367939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:48.368056 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:48.368523 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:48.368531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:48.368692 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:59:48.368811 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:59:48.369006 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":70,"thread_start_us":78,"threads_started":1}
I20260926 06:59:48.369246 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:48.357228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:40191 (local address 127.1.74.65:53045)
0926 06:59:48.357527 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:48.357538 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:59:48.357601 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:59:48.358466 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:59:48.358469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:59:48.358989 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:59:48.359283 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:48.359291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:48.360889 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:48.360889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:48.362205 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:59:48.362207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:59:48.362937 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:59:48.362938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:59:48.363056 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:59:48.363739 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:59:48.363743 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:59:48.365595 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:59:48.367581 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:48.367582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:48.367582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:48.367836 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:48.368066 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:59:48.368066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:59:48.368067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:59:48.368129 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:59:48.368523 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:59:48.368531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:59:48.368692 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:59:48.368811 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:59:48.369060 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":198,"thread_start_us":77,"threads_started":1}
W20260926 06:59:48.370069 16173 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
W20260926 06:59:48.370172 16175 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
I20260926 06:59:48.370235 16175 leader_election.cc:302] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 444edb1aa920424395882c1f96d49299; no voters: 2182d3ae26574f9caedcdf253178c5cd, a5027532a86d4155b0c79e89dc7bde51
I20260926 06:59:48.370332 16560 raft_consensus.cc:2726] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:59:48.615515 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:48.615382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:34833)
0926 06:59:48.615446 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:48.615476 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 06:59:48.676839 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:48.676644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:59439)
0926 06:59:48.676754 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:48.676798 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260926 06:59:48.798482 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:48.798274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:56205)
0926 06:59:48.798402 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:48.798444 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260926 06:59:49.616237 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:49.616015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:49037)
0926 06:59:49.616138 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:49.616175 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260926 06:59:49.677457 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:49.677267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:45171)
0926 06:59:49.677377 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:49.677413 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260926 06:59:49.799119 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:49.798931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:44223)
0926 06:59:49.799052 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:49.799084 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260926 06:59:50.230513 16570 raft_consensus.cc:482] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:59:50.230651 16570 raft_consensus.cc:504] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:50.230849 16570 leader_election.cc:288] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523), 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191)
W20260926 06:59:50.231374 16175 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
W20260926 06:59:50.231472 16173 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
I20260926 06:59:50.231542 16173 leader_election.cc:302] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 444edb1aa920424395882c1f96d49299; no voters: 2182d3ae26574f9caedcdf253178c5cd, a5027532a86d4155b0c79e89dc7bde51
I20260926 06:59:50.231632 16570 raft_consensus.cc:2726] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:59:50.617085 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:50.616808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:44613)
0926 06:59:50.616951 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:50.617004 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260926 06:59:50.678071 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:50.677857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:51895)
0926 06:59:50.677986 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:50.678019 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260926 06:59:50.799738 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:50.799551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:37347)
0926 06:59:50.799664 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:50.799697 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260926 06:59:51.617918 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:51.617705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:45045)
0926 06:59:51.617835 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:51.617871 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260926 06:59:51.678730 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:51.678543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:51921)
0926 06:59:51.678655 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:51.678690 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260926 06:59:51.800391 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:51.800183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:41627)
0926 06:59:51.800317 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:51.800353 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260926 06:59:52.618559 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:52.618364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:59015)
0926 06:59:52.618479 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:52.618513 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260926 06:59:52.679461 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:52.679230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:42763)
0926 06:59:52.679383 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:52.679418 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260926 06:59:52.801173 16579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:52.800929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:39591)
0926 06:59:52.801091 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:52.801130 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260926 06:59:53.272339 16580 raft_consensus.cc:482] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:59:53.272435 16580 raft_consensus.cc:504] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:53.272608 16580 leader_election.cc:288] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523), 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191)
W20260926 06:59:53.272993 16173 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
W20260926 06:59:53.273063 16175 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
I20260926 06:59:53.273144 16175 leader_election.cc:302] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 444edb1aa920424395882c1f96d49299; no voters: 2182d3ae26574f9caedcdf253178c5cd, a5027532a86d4155b0c79e89dc7bde51
I20260926 06:59:53.273243 16580 raft_consensus.cc:2726] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:59:53.619354 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:53.619070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:34285)
0926 06:59:53.619205 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:53.619246 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260926 06:59:53.680086 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:53.679871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:38243)
0926 06:59:53.679999 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:53.680032 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260926 06:59:53.801863 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:53.801632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:55331)
0926 06:59:53.801780 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:53.801819 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260926 06:59:54.620007 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:54.619791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:53109)
0926 06:59:54.619915 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:54.619959 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260926 06:59:54.680688 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:54.680507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:54433)
0926 06:59:54.680614 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:54.680648 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260926 06:59:54.802647 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:54.802405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:54815)
0926 06:59:54.802563 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:54.802608 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260926 06:59:55.223479 16587 raft_consensus.cc:482] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:59:55.223572 16587 raft_consensus.cc:504] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:55.223773 16587 leader_election.cc:288] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523), 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191)
W20260926 06:59:55.224370 16173 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
W20260926 06:59:55.224464 16175 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
I20260926 06:59:55.224567 16175 leader_election.cc:302] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 444edb1aa920424395882c1f96d49299; no voters: 2182d3ae26574f9caedcdf253178c5cd, a5027532a86d4155b0c79e89dc7bde51
I20260926 06:59:55.224660 16587 raft_consensus.cc:2726] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:59:55.620747 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:55.620505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:36993)
0926 06:59:55.620653 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:55.620696 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260926 06:59:55.681427 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:55.681179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:36283)
0926 06:59:55.681344 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:55.681382 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260926 06:59:55.803376 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:55.803110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:47995)
0926 06:59:55.803292 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:55.803332 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260926 06:59:56.621551 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:56.621274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:43759)
0926 06:59:56.621456 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:56.621502 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
I20260926 06:59:56.682094 16592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:56.681874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:47929)
0926 06:59:56.682008 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:56.682043 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260926 06:59:56.803985 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:56.803784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:41079)
0926 06:59:56.803911 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:56.803945 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260926 06:59:57.622303 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:57.622051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:49617)
0926 06:59:57.622222 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:57.622254 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":114,"threads_started":1}
I20260926 06:59:57.682713 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:57.682500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:49625)
0926 06:59:57.682627 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:57.682667 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260926 06:59:57.804611 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:57.804381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:45699)
0926 06:59:57.804526 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:57.804570 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260926 06:59:58.468525 16597 raft_consensus.cc:482] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:59:58.468619 16597 raft_consensus.cc:504] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5027532a86d4155b0c79e89dc7bde51" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 45523 } } peers { permanent_uuid: "2182d3ae26574f9caedcdf253178c5cd" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 40191 } } peers { permanent_uuid: "444edb1aa920424395882c1f96d49299" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44825 } }
I20260926 06:59:58.468829 16597 leader_election.cc:288] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523), 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191)
W20260926 06:59:58.469239 16173 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5027532a86d4155b0c79e89dc7bde51 (127.1.74.67:45523): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
W20260926 06:59:58.469297 16175 leader_election.cc:341] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2182d3ae26574f9caedcdf253178c5cd (127.1.74.66:40191): Not found: Tablet not found: 91f2c34048e9448fab46835d5c9fe0b3
I20260926 06:59:58.469327 16175 leader_election.cc:302] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 444edb1aa920424395882c1f96d49299; no voters: 2182d3ae26574f9caedcdf253178c5cd, a5027532a86d4155b0c79e89dc7bde51
I20260926 06:59:58.469386 16597 raft_consensus.cc:2726] T 91f2c34048e9448fab46835d5c9fe0b3 P 444edb1aa920424395882c1f96d49299 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:59:58.623008 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:58.622794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:54803)
0926 06:59:58.622928 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:58.622960 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260926 06:59:58.683414 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:58.683187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:56877)
0926 06:59:58.683336 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:58.683374 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":137,"threads_started":1}
I20260926 06:59:58.805295 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:58.805069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:46613)
0926 06:59:58.805207 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:58.805249 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260926 06:59:59.623703 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:59.623489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:51133)
0926 06:59:59.623619 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:59.623654 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260926 06:59:59.684020 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:59.683813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:39269)
0926 06:59:59.683935 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:59.683974 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260926 06:59:59.805956 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:59:59.805735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:40941)
0926 06:59:59.805871 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 06:59:59.805912 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260926 07:00:00.624461 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:00.624225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:37715)
0926 07:00:00.624373 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:00.624413 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260926 07:00:00.684767 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:00.684494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:52709)
0926 07:00:00.684659 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:00.684702 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260926 07:00:00.806663 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:00.806437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:51249)
0926 07:00:00.806571 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:00.806609 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260926 07:00:01.625471 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:01.625131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.65:45625)
0926 07:00:01.625335 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:01.625376 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":59,"threads_started":1}
I20260926 07:00:01.685491 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:01.685305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.66:37029)
0926 07:00:01.685418 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:01.685448 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260926 07:00:01.805837 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 16128
I20260926 07:00:01.807631 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:01.807293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42169 (local address 127.1.74.67:49627)
0926 07:00:01.807537 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:01.807577 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":56,"threads_started":1}
I20260926 07:00:01.810688 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 16292
I20260926 07:00:01.814950 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 16424
2026-09-26T07:00:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18186 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:00:01 dist-test-slave-sfvt krb5kdc[16612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:00:01 dist-test-slave-sfvt krb5kdc[16612](info): set up 2 sockets
Sep 26 07:00:01 dist-test-slave-sfvt krb5kdc[16612](info): commencing operation
krb5kdc: starting...
W20260926 07:00:02.855623 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:00:02 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:00:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:00:02.988582 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42423
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:00:03.071753 16625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:03.072080 16625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:03.072122 16625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:03.074307 16625 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:00:03.074371 16625 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:00:03.074395 16625 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:00:03.074416 16625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:00:03.074440 16625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:03.074463 16625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:03.074499 16625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:00:03.074527 16625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:00:03.077374 16625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:03.077675 16625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:03.077965 16625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:03.083813 16631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:03.083927 16625 server_base.cc:1034] running on GCE node
W20260926 07:00:03.083818 16630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:03.084067 16633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:03.084429 16625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:03.084811 16625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:03.086009 16625 hybrid_clock.cc:648] HybridClock initialized: now 1790406003085988 us; error 36 us; skew 500 ppm
Sep 26 07:00:03 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:03.088653 16625 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:00:03.089098 16625 webserver.cc:460] Webserver started at http://127.1.74.126:45939/ using document root <none> and password file <none>
I20260926 07:00:03.089344 16625 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:03.089412 16625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:03.089551 16625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:00:03.090632 16625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "3de3469878b44214939f5a73bda745ac"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "a1e617bf76008ffe26b5782b0236c937"
server_key_iv: "6b7cd3e259eff9d61fa78cdab08d424f"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.090986 16625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "3de3469878b44214939f5a73bda745ac"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "a1e617bf76008ffe26b5782b0236c937"
server_key_iv: "6b7cd3e259eff9d61fa78cdab08d424f"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.092465 16625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:00:03.093312 16640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.093485 16625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:03.093585 16625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "3de3469878b44214939f5a73bda745ac"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "a1e617bf76008ffe26b5782b0236c937"
server_key_iv: "6b7cd3e259eff9d61fa78cdab08d424f"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.093667 16625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:03.110363 16625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:03.112852 16625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:03.113044 16625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:03.117856 16625 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:42423
I20260926 07:00:03.117921 16692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:42423 every 8 connection(s)
I20260926 07:00:03.118337 16625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:00:03.119210 16693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:00:03.121615 16693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Bootstrap starting.
I20260926 07:00:03.122227 16693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Neither blocks nor log segments found. Creating new log.
I20260926 07:00:03.122539 16693 log.cc:824] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:03.123416 16693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: No bootstrap required, opened a new log
I20260926 07:00:03.125057 16693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:03.125227 16693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:00:03.125303 16693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de3469878b44214939f5a73bda745ac, State: Initialized, Role: FOLLOWER
I20260926 07:00:03.125515 16693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:03.125610 16693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:00:03.125644 16693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:00:03.125717 16693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:00:03.126482 16693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:03.126600 16693 leader_election.cc:302] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de3469878b44214939f5a73bda745ac; no voters:
I20260926 07:00:03.126786 16693 leader_election.cc:288] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:00:03.126864 16696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:00:03.127043 16693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:00:03.127068 16696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 1 LEADER]: Becoming Leader. State: Replica: 3de3469878b44214939f5a73bda745ac, State: Running, Role: LEADER
I20260926 07:00:03.127243 16696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:03.127633 16697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3de3469878b44214939f5a73bda745ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } } }
I20260926 07:00:03.127710 16697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: This master's current role is: LEADER
I20260926 07:00:03.127875 16698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de3469878b44214939f5a73bda745ac. Latest consensus state: current_term: 1 leader_uuid: "3de3469878b44214939f5a73bda745ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } } }
I20260926 07:00:03.127894 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 16625
I20260926 07:00:03.127928 16698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: This master's current role is: LEADER
I20260926 07:00:03.128005 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:00:03.128230 1321 external_mini_cluster.cc:1428] Setting key 8bcc3d955c2aa5d40c9f5201281ce31d
I20260926 07:00:03.128357 16701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:00:03.128940 16701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:00:03.130497 16701 catalog_manager.cc:1347] Generated new cluster ID: 0183602ff89f430783bb522bb60c8723
I20260926 07:00:03.130541 16701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:00:03 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:00:03.138494 16701 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:00:03.139245 16701 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:00:03.142839 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.128732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.1.74.126:42423)
0926 07:00:03.128889 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.128906 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.128965 (+ 59us) server_negotiation.cc:407] Connection header received
0926 07:00:03.129097 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.129105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.129170 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.129332 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:03.130103 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.131726 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.132917 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.133643 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.133755 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.138271 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:03.138288 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:03.138288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:03.138304 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:03.140602 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.141038 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.141041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.141042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.141166 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.141385 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.141385 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.141386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.141678 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:03.141818 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.142376 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.142555 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260926 07:00:03.149789 16701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Generated new TSK 0
I20260926 07:00:03.150020 16701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 07:00:03.187726 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42423
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:00:03.291066 16719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:03.291330 16719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:03.291393 16719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:03.292557 16719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:03.292595 16719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:03.292622 16719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 07:00:03.294076 16719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:03.294312 16719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:03.294564 16719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:03.300104 16724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:03.300361 16719 server_base.cc:1034] running on GCE node
W20260926 07:00:03.300065 16727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:03.300086 16725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:03.300711 16719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:03.300969 16719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:03.302127 16719 hybrid_clock.cc:648] HybridClock initialized: now 1790406003302116 us; error 30 us; skew 500 ppm
Sep 26 07:00:03 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:03.304808 16719 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 07:00:03.305346 16719 webserver.cc:460] Webserver started at http://127.1.74.65:39365/ using document root <none> and password file <none>
I20260926 07:00:03.305590 16719 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:03.305663 16719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:03.305807 16719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:00:03.306869 16719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "20f696558ad2452da61883f6fcc428ea"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "d21d6aa5f40c90bd83f8d4048b03c5df"
server_key_iv: "f252ca5c106c53ee30ab9fa8fbbc8d56"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.307226 16719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "20f696558ad2452da61883f6fcc428ea"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "d21d6aa5f40c90bd83f8d4048b03c5df"
server_key_iv: "f252ca5c106c53ee30ab9fa8fbbc8d56"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.308549 16719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:03.309384 16734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.309579 16719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:03.309728 16719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "20f696558ad2452da61883f6fcc428ea"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "d21d6aa5f40c90bd83f8d4048b03c5df"
server_key_iv: "f252ca5c106c53ee30ab9fa8fbbc8d56"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.309808 16719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:03.320211 16719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:03.322664 16719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:03.322834 16719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:03.323077 16719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:00:03.323472 16719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:00:03.323524 16719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.323565 16719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:00:03.323594 16719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.329522 16719 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:34517
I20260926 07:00:03.329624 16847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:34517 every 8 connection(s)
I20260926 07:00:03.329931 16719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 07:00:03.333639 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 16719
I20260926 07:00:03.333801 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 07:00:03.334008 1321 external_mini_cluster.cc:1428] Setting key f837408fde26ba97a9d2fe2ea129eff5
Sep 26 07:00:03 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:00:03.340449 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.330711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:55207 (local address 127.1.74.126:42423)
0926 07:00:03.330788 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.330791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.330948 (+ 157us) server_negotiation.cc:407] Connection header received
0926 07:00:03.331544 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.331545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.331560 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.331614 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:03.332574 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.333092 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.334488 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.334716 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.334807 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.337037 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:03.337039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:03.337039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:03.337047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:03.338795 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.339399 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.339400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.339401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.339453 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.339770 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.339771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.339771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.339878 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:03.339927 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.340314 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.340382 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:00:03.340742 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.330657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:55207)
0926 07:00:03.330822 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:03.330859 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:03.331447 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:03.331673 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:03.331677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:03.332267 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:00:03.332468 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.332473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.333198 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.333199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.334391 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.334394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.334819 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.334820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.334940 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.335522 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:00:03.335530 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:00:03.336963 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:00:03.338885 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:03.338892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:03.338894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:03.339323 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:03.339525 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:03.339527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:03.339527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:03.339680 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:03.339948 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:00:03.339956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:00:03.340157 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 07:00:03.340279 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 07:00:03.340469 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260926 07:00:03.341252 16848 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42423
I20260926 07:00:03.341367 16848 heartbeater.cc:464] Registering TS with master...
I20260926 07:00:03.341553 16848 heartbeater.cc:511] Master 127.1.74.126:42423 requested a full tablet report, sending...
I20260926 07:00:03.342202 16657 ts_manager.cc:194] Registered new tserver with Master: 20f696558ad2452da61883f6fcc428ea (127.1.74.65:34517)
I20260926 07:00:03.343173 16657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:55207
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 07:00:03.362577 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42423
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:00:03.443562 16853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:03.443758 16853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:03.443781 16853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:03.444979 16853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:03.445029 16853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:03.445055 16853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 07:00:03.446481 16853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:03.446725 16853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:03.446969 16853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:03.451942 16861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:03.451949 16859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:03.451979 16853 server_base.cc:1034] running on GCE node
W20260926 07:00:03.451978 16858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:03.452404 16853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:03.452597 16853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:03.453780 16853 hybrid_clock.cc:648] HybridClock initialized: now 1790406003453766 us; error 27 us; skew 500 ppm
Sep 26 07:00:03 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:03.456311 16853 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 07:00:03.456847 16853 webserver.cc:460] Webserver started at http://127.1.74.66:34613/ using document root <none> and password file <none>
I20260926 07:00:03.457091 16853 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:03.457159 16853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:03.457304 16853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:00:03.458343 16853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "18c867f7fbb9495f943b9a4a95cde8db"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "9716c9dcb511afd01cf0c5b76054b576"
server_key_iv: "b77b031ce83fad681f15379a2cf29f61"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.458702 16853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "18c867f7fbb9495f943b9a4a95cde8db"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "9716c9dcb511afd01cf0c5b76054b576"
server_key_iv: "b77b031ce83fad681f15379a2cf29f61"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.460067 16853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:00:03.460748 16868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.460917 16853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:00:03.461015 16853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "18c867f7fbb9495f943b9a4a95cde8db"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "9716c9dcb511afd01cf0c5b76054b576"
server_key_iv: "b77b031ce83fad681f15379a2cf29f61"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.461095 16853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:03.473819 16853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:03.475710 16853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:03.475867 16853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:03.476096 16853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:00:03.476401 16853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:00:03.476456 16853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.476498 16853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:00:03.476526 16853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.482375 16853 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:41377
I20260926 07:00:03.482447 16981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:41377 every 8 connection(s)
I20260926 07:00:03.482777 16853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 07:00:03.487216 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 16853
I20260926 07:00:03.487376 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 07:00:03.487615 1321 external_mini_cluster.cc:1428] Setting key bd3ce3f69f3b85fa36daef9d4a7e9f5c
Sep 26 07:00:03 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:00:03.492770 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.483537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:59999 (local address 127.1.74.126:42423)
0926 07:00:03.483580 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.483582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.483813 (+ 231us) server_negotiation.cc:407] Connection header received
0926 07:00:03.484338 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.484339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.484351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.484396 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:03.485327 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.485837 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.486848 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.487079 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.487166 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.489850 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:03.489852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:03.489852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:03.489860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:03.491391 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.491806 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.491807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.491807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.491868 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.492124 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.492124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.492125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.492246 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:03.492323 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.492596 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.492674 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:00:03.493012 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.483504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:59999)
0926 07:00:03.483689 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:03.483729 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:03.484234 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:03.484468 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:03.484472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:03.485034 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:00:03.485227 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.485232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.485946 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.485947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.486754 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.486756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.487178 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.487179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.487380 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.487975 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:00:03.487984 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:00:03.489693 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:00:03.491485 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:03.491491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:03.491494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:03.491728 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:03.491935 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:03.491936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:03.491936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:03.492049 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:03.492314 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:00:03.492320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:00:03.492462 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 07:00:03.492567 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:00:03.492742 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260926 07:00:03.493505 16982 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42423
I20260926 07:00:03.493613 16982 heartbeater.cc:464] Registering TS with master...
I20260926 07:00:03.493827 16982 heartbeater.cc:511] Master 127.1.74.126:42423 requested a full tablet report, sending...
I20260926 07:00:03.494210 16657 ts_manager.cc:194] Registered new tserver with Master: 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377)
I20260926 07:00:03.494741 16657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:59999
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 07:00:03.514463 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42423
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:00:03.595052 16987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:03.595276 16987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:03.595414 16987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:03.596760 16987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:03.596808 16987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:03.596836 16987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 07:00:03.598361 16987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:03.598555 16987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:03.598737 16987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:00:03.604012 16987 server_base.cc:1034] running on GCE node
W20260926 07:00:03.604041 16995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:03.604161 16992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:03.604027 16993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:03.604429 16987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:03.604679 16987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:03.605868 16987 hybrid_clock.cc:648] HybridClock initialized: now 1790406003605857 us; error 44 us; skew 500 ppm
Sep 26 07:00:03 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:03.608400 16987 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 07:00:03.608847 16987 webserver.cc:460] Webserver started at http://127.1.74.67:38855/ using document root <none> and password file <none>
I20260926 07:00:03.609081 16987 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:03.609148 16987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:03.609299 16987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:00:03.610373 16987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "cdd9d81a566474d3f4204e3dd7484ed3"
server_key_iv: "5d5f8753eb7241638aa11b9e8c34c818"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.610735 16987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "cdd9d81a566474d3f4204e3dd7484ed3"
server_key_iv: "5d5f8753eb7241638aa11b9e8c34c818"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.612178 16987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:00:03.613025 17002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.613194 16987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:00:03.613292 16987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "cdd9d81a566474d3f4204e3dd7484ed3"
server_key_iv: "5d5f8753eb7241638aa11b9e8c34c818"
server_key_version: "encryptionkey@0"
I20260926 07:00:03.613370 16987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:03.621030 16987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:03.623328 16987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:03.623492 16987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:03.623725 16987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:00:03.624029 16987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:00:03.624080 16987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.624121 16987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:00:03.624150 16987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:03.630574 16987 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:37997
I20260926 07:00:03.630646 17115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:37997 every 8 connection(s)
I20260926 07:00:03.630973 16987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:00:03 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:00:03.639493 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 16987
I20260926 07:00:03.639591 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 07:00:03.639824 1321 external_mini_cluster.cc:1428] Setting key e7f3f2307c4e5ef9de0a6417fd6264f9
I20260926 07:00:03.639935 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.631674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:34913 (local address 127.1.74.126:42423)
0926 07:00:03.631716 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.631718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.631914 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:00:03.632372 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.632373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.632384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.632435 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:03.633339 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.633845 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.634810 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.635002 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.635060 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.637263 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:03.637264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:03.637264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:03.637271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:03.638520 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.638913 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.638914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.638914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.638965 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.639178 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.639179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.639179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.639373 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:03.639454 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.639756 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.639839 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 07:00:03.640175 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.631661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:34913)
0926 07:00:03.631792 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:03.631831 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:03.632291 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:03.632487 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:03.632491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:03.633057 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:00:03.633250 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.633255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.633931 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.633931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.634725 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.634727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.635083 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.635084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.635190 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.635782 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:00:03.635790 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:00:03.637165 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:00:03.638603 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:03.638609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:03.638611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:03.638848 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:03.639023 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:03.639024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:03.639024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:03.639127 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:03.639442 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:00:03.639446 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:00:03.639586 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 07:00:03.639725 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 07:00:03.639903 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260926 07:00:03.640588 17116 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42423
I20260926 07:00:03.640708 17116 heartbeater.cc:464] Registering TS with master...
I20260926 07:00:03.640887 17116 heartbeater.cc:511] Master 127.1.74.126:42423 requested a full tablet report, sending...
I20260926 07:00:03.641225 16656 ts_manager.cc:194] Registered new tserver with Master: 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997)
I20260926 07:00:03.641620 16656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:34913
I20260926 07:00:03.651633 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:00:03.658545 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.652918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.1.74.126:42423)
0926 07:00:03.652988 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.652990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.653051 (+ 61us) server_negotiation.cc:407] Connection header received
0926 07:00:03.653095 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.653096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.653105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.653141 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:03.653638 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.654121 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.654807 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.655002 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.655051 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.656306 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:03.656307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:03.656308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:03.656313 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:03.657518 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.657861 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.657862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.657862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.657911 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:03.658069 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:03.658069 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:03.658070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:03.658202 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:03.658275 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.658377 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.658435 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:00:03.660439 16656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:00:03.661995 16656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:00:03.668466 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.664488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:37997 (local address 127.0.0.1:43484)
0926 07:00:03.664746 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:03.664757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:03.664798 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:03.665222 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:03.665222 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:03.665224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:00:03.665269 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.665270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.666494 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.666495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.667473 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.667476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.668159 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.668160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.668237 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.668252 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 07:00:03.668330 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:00:03.668386 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":66,"threads_started":1}
I20260926 07:00:03.668566 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.664404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.1.74.66:41377)
0926 07:00:03.664593 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.664598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.664609 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:00:03.664766 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.664771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.664788 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.664843 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:00:03.665120 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.666259 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.667583 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.668227 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.668289 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.668337 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.668434 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.668483 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
I20260926 07:00:03.668735 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.664456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.1.74.65:34517)
0926 07:00:03.664680 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.664684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.664700 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:00:03.664841 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.664844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.664861 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.664945 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:00:03.665220 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.665939 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.667449 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.668177 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.668249 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.668306 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.668606 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.668650 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260926 07:00:03.669078 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.664687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.1.74.67:37997)
0926 07:00:03.664999 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.665005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.665015 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:00:03.665054 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.665056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.665072 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.665141 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:00:03.665354 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.666385 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.667578 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.668066 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.668157 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.668227 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.668987 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.669021 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":74,"threads_started":1}
I20260926 07:00:03.668563 17127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.664299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:41377 (local address 127.0.0.1:46626)
0926 07:00:03.664496 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:03.664541 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:03.664630 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:03.664977 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:03.664978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:03.664979 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:00:03.665031 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.665032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.666359 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.666360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.667464 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.667465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.668289 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.668289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.668359 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.668361 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:00:03.668418 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 07:00:03.668451 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260926 07:00:03.668872 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.664340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:34517 (local address 127.0.0.1:53484)
0926 07:00:03.664654 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:03.664664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:03.664714 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:03.664949 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:03.664952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:03.664954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:00:03.665031 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.665035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.666038 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.666039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.667355 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.667358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.668353 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.668354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.668423 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.668426 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:00:03.668475 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 07:00:03.668512 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":46,"thread_start_us":88,"threads_started":1}
I20260926 07:00:03.669902 16916 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc07a78b4ca4a8ebddd038e184097f7 (DEFAULT_TABLE table=test-table [id=405dc8bd33b54024878b5d9929fdd476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:00:03.669899 17050 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc07a78b4ca4a8ebddd038e184097f7 (DEFAULT_TABLE table=test-table [id=405dc8bd33b54024878b5d9929fdd476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:00:03.669978 16782 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc07a78b4ca4a8ebddd038e184097f7 (DEFAULT_TABLE table=test-table [id=405dc8bd33b54024878b5d9929fdd476]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:00:03.670236 16916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc07a78b4ca4a8ebddd038e184097f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:00:03.670238 17050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc07a78b4ca4a8ebddd038e184097f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:00:03.670243 16782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc07a78b4ca4a8ebddd038e184097f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:00:03.672986 17133 tablet_bootstrap.cc:492] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: Bootstrap starting.
I20260926 07:00:03.673460 17135 tablet_bootstrap.cc:492] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Bootstrap starting.
I20260926 07:00:03.673676 17134 tablet_bootstrap.cc:492] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Bootstrap starting.
I20260926 07:00:03.673774 17133 tablet_bootstrap.cc:654] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: Neither blocks nor log segments found. Creating new log.
I20260926 07:00:03.674420 17134 tablet_bootstrap.cc:654] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Neither blocks nor log segments found. Creating new log.
I20260926 07:00:03.674398 17133 log.cc:824] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:03.675029 17134 log.cc:824] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:03.675108 17133 tablet_bootstrap.cc:492] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: No bootstrap required, opened a new log
I20260926 07:00:03.675230 17133 ts_tablet_manager.cc:1397] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:00:03.675338 17135 tablet_bootstrap.cc:654] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Neither blocks nor log segments found. Creating new log.
I20260926 07:00:03.676023 17134 tablet_bootstrap.cc:492] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: No bootstrap required, opened a new log
I20260926 07:00:03.676030 17135 log.cc:824] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:03.676157 17134 ts_tablet_manager.cc:1397] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 07:00:03.676779 17135 tablet_bootstrap.cc:492] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: No bootstrap required, opened a new log
I20260926 07:00:03.676851 17135 ts_tablet_manager.cc:1397] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 07:00:03.677033 17133 raft_consensus.cc:348] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:03.677199 17133 raft_consensus.cc:374] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:00:03.677259 17133 raft_consensus.cc:729] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c867f7fbb9495f943b9a4a95cde8db, State: Initialized, Role: FOLLOWER
I20260926 07:00:03.677397 17133 consensus_queue.cc:260] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:03.677696 17133 ts_tablet_manager.cc:1428] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:00:03.677830 16982 heartbeater.cc:503] Master 127.1.74.126:42423 was elected leader, sending a full tablet report...
I20260926 07:00:03.678133 17134 raft_consensus.cc:348] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:03.678303 17134 raft_consensus.cc:374] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:00:03.678373 17134 raft_consensus.cc:729] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587d36d0f2cb4ac7a1cb62dc1c8b45ec, State: Initialized, Role: FOLLOWER
I20260926 07:00:03.678508 17134 consensus_queue.cc:260] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:03.678817 17134 ts_tablet_manager.cc:1428] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 07:00:03.678892 17116 heartbeater.cc:503] Master 127.1.74.126:42423 was elected leader, sending a full tablet report...
I20260926 07:00:03.678923 17135 raft_consensus.cc:348] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:03.679064 17135 raft_consensus.cc:374] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:00:03.679121 17135 raft_consensus.cc:729] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f696558ad2452da61883f6fcc428ea, State: Initialized, Role: FOLLOWER
I20260926 07:00:03.679291 17135 consensus_queue.cc:260] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:03.679603 17135 ts_tablet_manager.cc:1428] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:00:03.679654 16848 heartbeater.cc:503] Master 127.1.74.126:42423 was elected leader, sending a full tablet report...
I20260926 07:00:03.719201 17141 raft_consensus.cc:482] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:00:03.719349 17141 raft_consensus.cc:504] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:03.719662 17141 leader_election.cc:288] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377), 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997)
I20260926 07:00:03.723307 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.719965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:54557 (local address 127.1.74.66:41377)
0926 07:00:03.720019 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.720021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.720034 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:00:03.720066 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.720068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.720081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.720170 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:00:03.720364 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.721122 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.722511 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.722916 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.722972 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.722981 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.723153 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.723204 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:00:03.723340 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.719871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:41377 (local address 127.1.74.65:54557)
0926 07:00:03.719935 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:03.719948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:03.719994 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:03.720195 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:03.720196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:03.720199 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:00:03.720273 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.720275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.721226 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.721227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.722429 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.722432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.722998 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.722999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.723081 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.723084 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:00:03.723145 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:00:03.723177 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":47}
I20260926 07:00:03.723358 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.720105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:52185 (local address 127.1.74.67:37997)
0926 07:00:03.720167 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.720169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.720236 (+ 67us) server_negotiation.cc:407] Connection header received
0926 07:00:03.720297 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.720298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.720307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.720380 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:00:03.720579 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.721280 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.722548 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.722977 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.723047 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.723058 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.723229 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.723296 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:00:03.723361 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.720020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:37997 (local address 127.1.74.65:52185)
0926 07:00:03.720179 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:03.720189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:03.720239 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:03.720398 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:03.720399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:03.720403 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:00:03.720455 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.720456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.721365 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.721366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.722451 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:03.722454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.723064 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.723065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:03.723148 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.723152 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:00:03.723207 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 07:00:03.723242 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":114,"threads_started":1}
I20260926 07:00:03.723686 17070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc07a78b4ca4a8ebddd038e184097f7" candidate_uuid: "20f696558ad2452da61883f6fcc428ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" is_pre_election: true
I20260926 07:00:03.723732 16936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc07a78b4ca4a8ebddd038e184097f7" candidate_uuid: "20f696558ad2452da61883f6fcc428ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c867f7fbb9495f943b9a4a95cde8db" is_pre_election: true
I20260926 07:00:03.723846 16936 raft_consensus.cc:2454] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f696558ad2452da61883f6fcc428ea in term 0.
I20260926 07:00:03.723847 17070 raft_consensus.cc:2454] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20f696558ad2452da61883f6fcc428ea in term 0.
I20260926 07:00:03.724041 16737 leader_election.cc:302] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c867f7fbb9495f943b9a4a95cde8db, 20f696558ad2452da61883f6fcc428ea; no voters:
I20260926 07:00:03.724166 17141 raft_consensus.cc:2781] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:00:03.724222 17141 raft_consensus.cc:482] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:00:03.724251 17141 raft_consensus.cc:3037] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:00:03.724953 17141 raft_consensus.cc:504] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:03.725078 17141 leader_election.cc:288] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 1 election: Requested vote from peers 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377), 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997)
I20260926 07:00:03.725242 17070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc07a78b4ca4a8ebddd038e184097f7" candidate_uuid: "20f696558ad2452da61883f6fcc428ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec"
I20260926 07:00:03.725251 16936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc07a78b4ca4a8ebddd038e184097f7" candidate_uuid: "20f696558ad2452da61883f6fcc428ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c867f7fbb9495f943b9a4a95cde8db"
I20260926 07:00:03.725325 16936 raft_consensus.cc:3037] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:00:03.725333 17070 raft_consensus.cc:3037] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:00:03.726191 16936 raft_consensus.cc:2454] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f696558ad2452da61883f6fcc428ea in term 1.
I20260926 07:00:03.726191 17070 raft_consensus.cc:2454] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f696558ad2452da61883f6fcc428ea in term 1.
I20260926 07:00:03.726496 16737 leader_election.cc:302] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c867f7fbb9495f943b9a4a95cde8db, 20f696558ad2452da61883f6fcc428ea; no voters:
I20260926 07:00:03.726590 17141 raft_consensus.cc:2781] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:00:03.726754 17141 raft_consensus.cc:686] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 1 LEADER]: Becoming Leader. State: Replica: 20f696558ad2452da61883f6fcc428ea, State: Running, Role: LEADER
I20260926 07:00:03.726897 17141 consensus_queue.cc:237] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:03.727809 16657 catalog_manager.cc:5495] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 20f696558ad2452da61883f6fcc428ea (127.1.74.65). New cstate: current_term: 1 leader_uuid: "20f696558ad2452da61883f6fcc428ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } health_report { overall_health: HEALTHY } } }
W20260926 07:00:03.733290 16983 tablet.cc:2369] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:00:03.741324 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.739107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.1.74.65:34517)
0926 07:00:03.739151 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:03.739153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:03.739210 (+ 57us) server_negotiation.cc:407] Connection header received
0926 07:00:03.739319 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:03.739320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:03.739342 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:03.739414 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:00:03.739519 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.739985 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.740685 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:03.740886 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:03.740939 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:03.741068 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:00:03.741163 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:00:03.741223 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:03.741245 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 07:00:03.741261 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:00:03.743212 17070 raft_consensus.cc:1261] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 1 FOLLOWER]: Refusing update from remote peer 20f696558ad2452da61883f6fcc428ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:00:03.743212 16936 raft_consensus.cc:1261] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [term 1 FOLLOWER]: Refusing update from remote peer 20f696558ad2452da61883f6fcc428ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:00:03.743618 17143 consensus_queue.cc:1035] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:00:03.743760 17141 consensus_queue.cc:1035] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:00:03.746886 17148 mvcc.cc:205] Tried to move back new op lower bound from 7333502991328759808 to 7333502991265992704. Current Snapshot: MvccSnapshot[applied={T|T < 7333502991328759808}]
I20260926 07:00:03.747417 17147 mvcc.cc:205] Tried to move back new op lower bound from 7333502991328759808 to 7333502991265992704. Current Snapshot: MvccSnapshot[applied={T|T < 7333502991328759808 or (T in {7333502991328759808})}]
I20260926 07:00:03.748299 16656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:00:03.748416 16656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:00:03.749154 16656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Sending DeleteTablet for 3 replicas of tablet 4cc07a78b4ca4a8ebddd038e184097f7
I20260926 07:00:03.749471 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 16719
I20260926 07:00:03.749585 16916 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc07a78b4ca4a8ebddd038e184097f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:00:03 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:46626
I20260926 07:00:03.749586 17050 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc07a78b4ca4a8ebddd038e184097f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:00:03 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:43484
I20260926 07:00:03.750975 17155 tablet_replica.cc:330] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: stopping tablet replica
I20260926 07:00:03.751080 17155 raft_consensus.cc:2229] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:00:03.751178 17155 raft_consensus.cc:2258] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:00:03.751633 17155 ts_tablet_manager.cc:1905] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:00:03.753726 17155 ts_tablet_manager.cc:1918] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:00:03.753808 17155 log.cc:1196] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/4cc07a78b4ca4a8ebddd038e184097f7
I20260926 07:00:03.754079 17155 ts_tablet_manager.cc:1939] T 4cc07a78b4ca4a8ebddd038e184097f7 P 18c867f7fbb9495f943b9a4a95cde8db: Deleting consensus metadata
I20260926 07:00:03.754451 16643 catalog_manager.cc:4841] TS 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377): tablet 4cc07a78b4ca4a8ebddd038e184097f7 (table test-table [id=405dc8bd33b54024878b5d9929fdd476]) successfully deleted
I20260926 07:00:03.755406 17156 tablet_replica.cc:330] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: stopping tablet replica
W20260926 07:00:03.755422 16641 connection.cc:535] client connection to 127.1.74.65:34517 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 07:00:03.755507 16641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:00:03.755515 17156 raft_consensus.cc:2229] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:00:03.755620 17156 raft_consensus.cc:2258] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:00:03.755722 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 16853
I20260926 07:00:03.756458 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:03.756341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:34517 (local address 127.0.0.1:53510)
0926 07:00:03.756385 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:03.756418 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 07:00:03.756494 17156 ts_tablet_manager.cc:1905] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:00:03.756592 16641 catalog_manager.cc:4584] TS 20f696558ad2452da61883f6fcc428ea (127.1.74.65:34517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cc07a78b4ca4a8ebddd038e184097f7: Network error: Client connection negotiation failed: client connection to 127.1.74.65:34517: connect: Connection refused (error 111)
I20260926 07:00:03.758571 17156 ts_tablet_manager.cc:1918] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:00:03.758651 17156 log.cc:1196] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/4cc07a78b4ca4a8ebddd038e184097f7
I20260926 07:00:03.758916 17156 ts_tablet_manager.cc:1939] T 4cc07a78b4ca4a8ebddd038e184097f7 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Deleting consensus metadata
I20260926 07:00:03.759527 16643 catalog_manager.cc:4841] TS 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997): tablet 4cc07a78b4ca4a8ebddd038e184097f7 (table test-table [id=405dc8bd33b54024878b5d9929fdd476]) successfully deleted
I20260926 07:00:03.762248 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 16987
I20260926 07:00:03.766681 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 16625
I20260926 07:00:23.772182 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42423
--webserver_interface=127.1.74.126
--webserver_port=45939
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:00:23.853087 17158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:23.853329 17158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:23.853366 17158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:23.854533 17158 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:00:23.854602 17158 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:00:23.854627 17158 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:00:23.854647 17158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:00:23.854678 17158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:23.854704 17158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:23.854735 17158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:00:23.854760 17158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:00:23.856240 17158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=45939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:23.856506 17158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:23.856730 17158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:23.865477 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:23.865499 17165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:23.865531 17158 server_base.cc:1034] running on GCE node
W20260926 07:00:23.865512 17164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:23.865981 17158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:23.866305 17158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:23.867470 17158 hybrid_clock.cc:648] HybridClock initialized: now 1790406023867457 us; error 49 us; skew 500 ppm
Sep 26 07:00:23 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:23.869833 17158 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:00:23.870271 17158 webserver.cc:460] Webserver started at http://127.1.74.126:45939/ using document root <none> and password file <none>
I20260926 07:00:23.870513 17158 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:23.870595 17158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:23.872005 17158 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:00:23.872670 17174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:23.872872 17158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:00:23.872978 17158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "3de3469878b44214939f5a73bda745ac"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "a1e617bf76008ffe26b5782b0236c937"
server_key_iv: "6b7cd3e259eff9d61fa78cdab08d424f"
server_key_version: "encryptionkey@0"
I20260926 07:00:23.873368 17158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:23.888173 17158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:23.889957 17158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:23.890111 17158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:23.894165 17158 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:42423
I20260926 07:00:23.894203 17226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:42423 every 8 connection(s)
I20260926 07:00:23.894613 17158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:00:23.896013 17227 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:00:23.896675 17227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Bootstrap starting.
I20260926 07:00:23.897094 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 17158
I20260926 07:00:23.897389 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:34517
--local_ip_for_outbound_sockets=127.1.74.65
--tserver_master_addrs=127.1.74.126:42423
--webserver_port=39365
--webserver_interface=127.1.74.65
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:00:23.905052 17227 log.cc:824] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:23.906898 17227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:00:23.907178 17227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Bootstrap complete.
I20260926 07:00:23.909449 17227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:23.909699 17227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de3469878b44214939f5a73bda745ac, State: Initialized, Role: FOLLOWER
I20260926 07:00:23.909884 17227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:23.909963 17227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:00:23.910053 17227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:00:23.910104 17227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:00:23.911090 17227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:23.911209 17227 leader_election.cc:302] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de3469878b44214939f5a73bda745ac; no voters:
I20260926 07:00:23.911420 17227 leader_election.cc:288] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:00:23.911454 17231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:00:23.911571 17231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 2 LEADER]: Becoming Leader. State: Replica: 3de3469878b44214939f5a73bda745ac, State: Running, Role: LEADER
I20260926 07:00:23.911686 17227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:00:23.911710 17231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:23.912016 17231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de3469878b44214939f5a73bda745ac. Latest consensus state: current_term: 2 leader_uuid: "3de3469878b44214939f5a73bda745ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } } }
I20260926 07:00:23.912092 17231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: This master's current role is: LEADER
I20260926 07:00:23.912179 17232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3de3469878b44214939f5a73bda745ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } } }
I20260926 07:00:23.912288 17237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:00:23.912309 17232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: This master's current role is: LEADER
I20260926 07:00:23.913137 17237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:00:23.913242 17237 catalog_manager.cc:1259] Loaded cluster ID: 0183602ff89f430783bb522bb60c8723
I20260926 07:00:23.913280 17237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:00:23.914402 17237 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:00:23.914578 17237 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Loaded TSK: 0
I20260926 07:00:23.928709 17237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Generated new TSK 1
I20260926 07:00:23.928819 17237 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:00:23.981169 17229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:23.981415 17229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:23.981463 17229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:23.982710 17229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:23.982784 17229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:23.982831 17229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 07:00:23.984297 17229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:34517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=39365
--tserver_master_addrs=127.1.74.126:42423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:23.984570 17229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:23.984807 17229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:23.989877 17254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:23.989933 17257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:23.989931 17255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:23.990280 17229 server_base.cc:1034] running on GCE node
I20260926 07:00:23.990499 17229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:23.990752 17229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:23.991909 17229 hybrid_clock.cc:648] HybridClock initialized: now 1790406023991886 us; error 41 us; skew 500 ppm
Sep 26 07:00:23 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:23.994526 17229 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 07:00:23.994990 17229 webserver.cc:460] Webserver started at http://127.1.74.65:39365/ using document root <none> and password file <none>
I20260926 07:00:23.995294 17229 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:23.995388 17229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:23.996760 17229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:23.997464 17264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:23.997664 17229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:23.997766 17229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "20f696558ad2452da61883f6fcc428ea"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "d21d6aa5f40c90bd83f8d4048b03c5df"
server_key_iv: "f252ca5c106c53ee30ab9fa8fbbc8d56"
server_key_version: "encryptionkey@0"
I20260926 07:00:23.998124 17229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:24.013794 17229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:24.015995 17229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:24.016189 17229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:24.016451 17229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:00:24.016902 17271 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:00:24.018869 17229 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:00:24.018932 17229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 07:00:24.018980 17229 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:00:24.019728 17229 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:00:24.019805 17229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 07:00:24.019910 17271 tablet_bootstrap.cc:492] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Bootstrap starting.
I20260926 07:00:24.027063 17229 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:34517
I20260926 07:00:24.027206 17378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:34517 every 8 connection(s)
I20260926 07:00:24.027547 17229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 07:00:24.030648 17271 log.cc:824] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:24.033397 17271 tablet_bootstrap.cc:492] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:00:24.033474 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 17229
I20260926 07:00:24.033718 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:41377
--local_ip_for_outbound_sockets=127.1.74.66
--tserver_master_addrs=127.1.74.126:42423
--webserver_port=34613
--webserver_interface=127.1.74.66
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:00:24.033751 17271 tablet_bootstrap.cc:492] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Bootstrap complete.
I20260926 07:00:24.035004 17271 ts_tablet_manager.cc:1397] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260926 07:00:24.036377 17271 raft_consensus.cc:348] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.036692 17271 raft_consensus.cc:729] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f696558ad2452da61883f6fcc428ea, State: Initialized, Role: FOLLOWER
Sep 26 07:00:24 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:00:24.036845 17271 consensus_queue.cc:260] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.037122 17271 ts_tablet_manager.cc:1428] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
I20260926 07:00:24.041990 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.028360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:33369 (local address 127.1.74.126:42423)
0926 07:00:24.029156 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.029163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.029191 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:00:24.029292 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.029296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.029334 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.029475 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:24.030116 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.031218 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.032307 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.032724 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.032845 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.037633 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:24.037638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:24.037638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:24.037650 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:24.039871 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.040428 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.040431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.040431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.040560 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.040881 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.040881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.040882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.041161 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:24.041240 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.041638 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.041787 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":78,"threads_started":1}
I20260926 07:00:24.042027 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.028257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:33369)
0926 07:00:24.028520 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:24.028568 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:24.029000 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:24.029502 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:24.029506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:24.029800 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:00:24.029990 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.029995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.031381 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.031382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.032189 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.032192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.032846 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.032847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.032995 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.035353 (+ 2358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:00:24.035364 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:00:24.037490 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:00:24.039963 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:24.039971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:24.039973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:24.040331 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:24.040642 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:24.040643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:24.040644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:24.040804 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:24.041239 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:00:24.041247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:00:24.041440 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 07:00:24.041572 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:00:24.041767 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
I20260926 07:00:24.043170 17379 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42423
I20260926 07:00:24.043311 17379 heartbeater.cc:464] Registering TS with master...
I20260926 07:00:24.043521 17379 heartbeater.cc:511] Master 127.1.74.126:42423 requested a full tablet report, sending...
I20260926 07:00:24.044363 17191 ts_manager.cc:194] Registered new tserver with Master: 20f696558ad2452da61883f6fcc428ea (127.1.74.65:34517)
I20260926 07:00:24.045553 17191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:33369
I20260926 07:00:24.046550 17379 heartbeater.cc:503] Master 127.1.74.126:42423 was elected leader, sending a full tablet report...
Sep 26 07:00:24 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.65@KRBTEST.COM
I20260926 07:00:24.054771 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.044988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:34517 (local address 127.0.0.1:59444)
0926 07:00:24.045374 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:24.045413 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:24.045476 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:24.045759 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:24.045764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:24.046088 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:00:24.046164 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.046166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.047543 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.047544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.048491 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.048495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.049128 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.049130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.049235 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.049776 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:00:24.049780 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:00:24.051465 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:00:24.053557 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:24.053564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:24.053566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:24.053824 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:24.054037 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:24.054038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:24.054038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:24.054105 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:24.054406 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:00:24.054412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:00:24.054465 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 07:00:24.054549 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 07:00:24.054664 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":87,"threads_started":1}
I20260926 07:00:24.054800 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.045224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.1.74.65:34517)
0926 07:00:24.045592 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.045597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.045605 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:00:24.045635 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.045636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.045649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.045704 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:24.046250 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.047452 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.048582 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.049034 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.049105 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.051563 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:24.051571 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:24.051571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:24.051578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:24.053475 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.053904 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.053907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.053907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.053965 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.054169 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.054170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.054170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.054331 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:24.054417 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.054598 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.054693 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":71,"threads_started":1}
I20260926 07:00:24.055540 17298 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc07a78b4ca4a8ebddd038e184097f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:00:03 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:59444
I20260926 07:00:24.056149 17388 tablet_replica.cc:330] stopping tablet replica
I20260926 07:00:24.056262 17388 raft_consensus.cc:2229] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:00:24.056329 17388 raft_consensus.cc:2258] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:00:24.056638 17388 ts_tablet_manager.cc:1905] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:00:24.059361 17388 ts_tablet_manager.cc:1918] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:00:24.059433 17388 log.cc:1196] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal/wals/4cc07a78b4ca4a8ebddd038e184097f7
I20260926 07:00:24.059667 17388 ts_tablet_manager.cc:1939] T 4cc07a78b4ca4a8ebddd038e184097f7 P 20f696558ad2452da61883f6fcc428ea: Deleting consensus metadata
I20260926 07:00:24.060345 17175 catalog_manager.cc:4841] TS 20f696558ad2452da61883f6fcc428ea (127.1.74.65:34517): tablet 4cc07a78b4ca4a8ebddd038e184097f7 (table test-table [id=405dc8bd33b54024878b5d9929fdd476]) successfully deleted
W20260926 07:00:24.121922 17384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:24.122126 17384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:24.122149 17384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:24.123239 17384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:24.123302 17384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:24.123337 17384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 07:00:24.124796 17384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:41377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=34613
--tserver_master_addrs=127.1.74.126:42423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:24.124980 17384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:24.125151 17384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:24.130141 17393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:24.130153 17394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:24.130271 17384 server_base.cc:1034] running on GCE node
W20260926 07:00:24.130162 17396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:24.130610 17384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:24.130856 17384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:24.132016 17384 hybrid_clock.cc:648] HybridClock initialized: now 1790406024131999 us; error 27 us; skew 500 ppm
Sep 26 07:00:24 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:24.134507 17384 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 07:00:24.134960 17384 webserver.cc:460] Webserver started at http://127.1.74.66:34613/ using document root <none> and password file <none>
I20260926 07:00:24.135135 17384 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:24.135200 17384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:24.136525 17384 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:00:24.137081 17403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:24.137262 17384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:00:24.137357 17384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "18c867f7fbb9495f943b9a4a95cde8db"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "9716c9dcb511afd01cf0c5b76054b576"
server_key_iv: "b77b031ce83fad681f15379a2cf29f61"
server_key_version: "encryptionkey@0"
I20260926 07:00:24.137686 17384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:24.149816 17384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:24.151830 17384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:24.151975 17384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:24.152220 17384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:00:24.152576 17384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:00:24.152611 17384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:24.152660 17384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:00:24.152689 17384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:24.158406 17384 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:41377
I20260926 07:00:24.158488 17516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:41377 every 8 connection(s)
I20260926 07:00:24.158821 17384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 07:00:24.159241 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 17384
I20260926 07:00:24.159587 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:37997
--local_ip_for_outbound_sockets=127.1.74.67
--tserver_master_addrs=127.1.74.126:42423
--webserver_port=38855
--webserver_interface=127.1.74.67
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:00:24 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:00:24.169485 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.159589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:46213 (local address 127.1.74.126:42423)
0926 07:00:24.159643 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.159645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.160101 (+ 456us) server_negotiation.cc:407] Connection header received
0926 07:00:24.160869 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.160870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.160901 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.160976 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:24.161986 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.162526 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.163614 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.163831 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.163919 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.166497 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:24.166499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:24.166499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:24.166507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:24.167942 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.168475 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.168475 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.168476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.168525 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.168825 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.168826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.168826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.168926 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:24.168972 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.169345 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.169417 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:00:24.169790 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.159579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:46213)
0926 07:00:24.159943 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:24.159993 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:24.160745 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:24.161062 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:24.161067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:24.161695 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:00:24.161893 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.161897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.162626 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.162627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.163524 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.163527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.163928 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.163929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.164052 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.164609 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:00:24.164618 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:00:24.166401 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:00:24.168028 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:24.168035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:24.168038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:24.168395 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:24.168593 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:24.168594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:24.168594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:24.168750 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:24.168990 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:00:24.168996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:00:24.169184 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 07:00:24.169322 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 07:00:24.169513 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":80,"threads_started":1}
I20260926 07:00:24.170359 17517 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42423
I20260926 07:00:24.170471 17517 heartbeater.cc:464] Registering TS with master...
I20260926 07:00:24.170665 17517 heartbeater.cc:511] Master 127.1.74.126:42423 requested a full tablet report, sending...
I20260926 07:00:24.171043 17191 ts_manager.cc:194] Registered new tserver with Master: 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377)
I20260926 07:00:24.171478 17191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:46213
W20260926 07:00:24.243095 17520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:24.243366 17520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:24.243408 17520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:24.244709 17520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:24.244773 17520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:24.244830 17520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 07:00:24.246340 17520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:37997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=38855
--tserver_master_addrs=127.1.74.126:42423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:24.246605 17520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:24.246851 17520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:24.251961 17528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:24.251961 17525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:24.251955 17526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:24.252202 17520 server_base.cc:1034] running on GCE node
I20260926 07:00:24.252404 17520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:24.252650 17520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:24.253818 17520 hybrid_clock.cc:648] HybridClock initialized: now 1790406024253801 us; error 39 us; skew 500 ppm
Sep 26 07:00:24 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:24.256484 17520 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 07:00:24.256942 17520 webserver.cc:460] Webserver started at http://127.1.74.67:38855/ using document root <none> and password file <none>
I20260926 07:00:24.257153 17520 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:24.257215 17520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:24.258522 17520 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:24.259039 17535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:24.259205 17520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:00:24.259339 17520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "cdd9d81a566474d3f4204e3dd7484ed3"
server_key_iv: "5d5f8753eb7241638aa11b9e8c34c818"
server_key_version: "encryptionkey@0"
I20260926 07:00:24.259652 17520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:24.277011 17520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:24.279026 17520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:24.279141 17520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:24.279384 17520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:00:24.279644 17520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:00:24.279673 17520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:24.279696 17520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:00:24.279708 17520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:24.285126 17520 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:37997
I20260926 07:00:24.285166 17648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:37997 every 8 connection(s)
I20260926 07:00:24.285537 17520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:00:24 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:00:24.294051 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.286281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:37503 (local address 127.1.74.126:42423)
0926 07:00:24.286353 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.286355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.286599 (+ 244us) server_negotiation.cc:407] Connection header received
0926 07:00:24.287065 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.287066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.287076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.287137 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:24.288134 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.288636 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.289579 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.289766 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.289815 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.291793 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:24.291795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:24.291795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:24.291801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:24.292940 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.293297 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.293298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.293298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.293345 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.293553 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.293553 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.293553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.293651 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:24.293700 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.293928 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.293988 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 07:00:24.294210 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.286269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:37503)
0926 07:00:24.286481 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:24.286514 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:24.286981 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:24.287210 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:24.287214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:24.287865 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:00:24.288054 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.288058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.288713 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.288714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.289499 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.289501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.289834 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.289835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.289931 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.290359 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:00:24.290366 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:00:24.291708 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:00:24.293009 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:24.293015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:24.293016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:24.293226 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:24.293402 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:24.293403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:24.293403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:24.293498 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:24.293705 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:00:24.293709 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:00:24.293821 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 07:00:24.293901 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:00:24.294014 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":108,"threads_started":1}
I20260926 07:00:24.294713 17649 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42423
I20260926 07:00:24.294834 17649 heartbeater.cc:464] Registering TS with master...
I20260926 07:00:24.294976 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 17520
I20260926 07:00:24.295006 17649 heartbeater.cc:511] Master 127.1.74.126:42423 requested a full tablet report, sending...
I20260926 07:00:24.295513 17191 ts_manager.cc:194] Registered new tserver with Master: 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997)
I20260926 07:00:24.295871 17191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:37503
I20260926 07:00:24.301348 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.295412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.1.74.126:42423)
0926 07:00:24.295459 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.295461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.295657 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:00:24.295731 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.295732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.295741 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.295783 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:24.296496 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.296950 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.297707 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.297910 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.297990 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.299047 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:24.299048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:24.299049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:24.299056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:24.300219 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.300572 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.300573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.300573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.300631 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.300838 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.300839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.300839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.300942 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:24.301129 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.301165 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.301235 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:00:24.302071 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:00:24.304447 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.302408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.1.74.126:42423)
0926 07:00:24.302457 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.302458 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.302543 (+ 85us) server_negotiation.cc:407] Connection header received
0926 07:00:24.302599 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.302600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.302624 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.302684 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:00:24.302756 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.303174 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.303912 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.304079 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.304129 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.304262 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:00:24.304307 (+ 45us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 07:00:24.304409 (+ 102us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59394: token expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 07:00:24.304510 17382 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59394: token expired
W20260926 07:00:24.304596 17122 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.74.126:42423)
I20260926 07:00:24.310058 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.304816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59406 (local address 127.1.74.126:42423)
0926 07:00:24.304856 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.304858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.304866 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:00:24.304886 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.304887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.304896 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.304949 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:24.305382 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.305816 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.306506 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.306692 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.306770 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.307852 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:24.307853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:24.307853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:24.307859 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:24.308924 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.309290 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.309291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.309292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.309352 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:24.309558 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:24.309559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:24.309559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:24.309664 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:24.309762 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.309889 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.309966 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:00:24.311056 17191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:00:24.312821 17191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:00:24.314875 17298 tablet_service.cc:1467] Processing CreateTablet for tablet 1de8ef40db8d43e390c53782427b4775 (DEFAULT_TABLE table=test-table [id=67f4c4b2fc3746f1958925f1361d8e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:00:24.315076 17298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de8ef40db8d43e390c53782427b4775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:00:24.316720 17271 tablet_bootstrap.cc:492] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Bootstrap starting.
I20260926 07:00:24.317065 17271 tablet_bootstrap.cc:654] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Neither blocks nor log segments found. Creating new log.
I20260926 07:00:24.318493 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.314701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.1.74.67:37997)
0926 07:00:24.314887 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.314892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.314901 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:00:24.314936 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.314938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.314951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.315201 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:00:24.315327 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.316180 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.317539 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.318154 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.318209 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.318249 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.318408 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.318435 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
I20260926 07:00:24.318503 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.314614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:37997 (local address 127.0.0.1:40894)
0926 07:00:24.314802 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:24.314811 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:24.314850 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:24.315053 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:24.315054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:24.315055 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:00:24.315110 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.315111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.316265 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.316266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.317449 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.317452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.318239 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.318240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.318311 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.318333 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:00:24.318391 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:00:24.318436 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":2}
I20260926 07:00:24.319242 17583 tablet_service.cc:1467] Processing CreateTablet for tablet 1de8ef40db8d43e390c53782427b4775 (DEFAULT_TABLE table=test-table [id=67f4c4b2fc3746f1958925f1361d8e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:00:24.319625 17583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de8ef40db8d43e390c53782427b4775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:00:24.319687 17271 tablet_bootstrap.cc:492] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: No bootstrap required, opened a new log
I20260926 07:00:24.319728 17271 ts_tablet_manager.cc:1397] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 07:00:24.319895 17271 raft_consensus.cc:348] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.319972 17271 raft_consensus.cc:374] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:00:24.320027 17271 raft_consensus.cc:729] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f696558ad2452da61883f6fcc428ea, State: Initialized, Role: FOLLOWER
I20260926 07:00:24.320075 17271 consensus_queue.cc:260] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.320174 17271 ts_tablet_manager.cc:1428] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:24.320391 17654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.314712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:41377 (local address 127.0.0.1:60140)
0926 07:00:24.316717 (+ 2005us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:24.316728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:24.316783 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:24.317007 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:24.317008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:24.317010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:00:24.317055 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.317056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.318265 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.318266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.319167 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.319169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.320138 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.320139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.320203 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.320208 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:00:24.320268 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:00:24.320317 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1992,"thread_start_us":56,"threads_started":1}
I20260926 07:00:24.320410 17656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.314904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.1.74.66:41377)
0926 07:00:24.315096 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.315100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.316842 (+ 1742us) server_negotiation.cc:407] Connection header received
0926 07:00:24.316887 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.316889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.316903 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.316990 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:00:24.317223 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.318159 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.319277 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.320030 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.320132 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.320192 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.320287 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.320329 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260926 07:00:24.321125 17451 tablet_service.cc:1467] Processing CreateTablet for tablet 1de8ef40db8d43e390c53782427b4775 (DEFAULT_TABLE table=test-table [id=67f4c4b2fc3746f1958925f1361d8e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:00:24.321420 17451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de8ef40db8d43e390c53782427b4775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:00:24.322738 17658 tablet_bootstrap.cc:492] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Bootstrap starting.
I20260926 07:00:24.323355 17658 tablet_bootstrap.cc:654] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Neither blocks nor log segments found. Creating new log.
I20260926 07:00:24.323946 17658 log.cc:824] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:24.324262 17659 tablet_bootstrap.cc:492] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: Bootstrap starting.
I20260926 07:00:24.324676 17658 tablet_bootstrap.cc:492] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: No bootstrap required, opened a new log
I20260926 07:00:24.324728 17658 ts_tablet_manager.cc:1397] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:00:24.324874 17659 tablet_bootstrap.cc:654] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: Neither blocks nor log segments found. Creating new log.
I20260926 07:00:24.325346 17659 log.cc:824] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:24.325985 17659 tablet_bootstrap.cc:492] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: No bootstrap required, opened a new log
I20260926 07:00:24.326064 17659 ts_tablet_manager.cc:1397] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:00:24.326171 17658 raft_consensus.cc:348] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.326313 17658 raft_consensus.cc:374] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:00:24.326335 17658 raft_consensus.cc:729] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587d36d0f2cb4ac7a1cb62dc1c8b45ec, State: Initialized, Role: FOLLOWER
I20260926 07:00:24.326484 17658 consensus_queue.cc:260] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.326776 17658 ts_tablet_manager.cc:1428] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:00:24.326807 17649 heartbeater.cc:503] Master 127.1.74.126:42423 was elected leader, sending a full tablet report...
I20260926 07:00:24.328056 17659 raft_consensus.cc:348] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.328203 17659 raft_consensus.cc:374] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:00:24.328248 17659 raft_consensus.cc:729] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c867f7fbb9495f943b9a4a95cde8db, State: Initialized, Role: FOLLOWER
I20260926 07:00:24.328368 17659 consensus_queue.cc:260] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.328611 17659 ts_tablet_manager.cc:1428] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:00:24.328666 17517 heartbeater.cc:503] Master 127.1.74.126:42423 was elected leader, sending a full tablet report...
W20260926 07:00:24.411576 17518 tablet.cc:2369] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:00:24.507617 17662 raft_consensus.cc:482] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:00:24.507742 17662 raft_consensus.cc:504] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.508183 17662 leader_election.cc:288] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377), 20f696558ad2452da61883f6fcc428ea (127.1.74.65:34517)
I20260926 07:00:24.511216 17656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.508488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:45947 (local address 127.1.74.66:41377)
0926 07:00:24.508555 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.508558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.508688 (+ 130us) server_negotiation.cc:407] Connection header received
0926 07:00:24.508717 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.508718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.508730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.508798 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:00:24.508982 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.509454 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.510502 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.510867 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.510923 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.510954 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.511097 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.511144 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:00:24.511291 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.508489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:33767 (local address 127.1.74.65:34517)
0926 07:00:24.508548 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.508550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.508558 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:00:24.508589 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.508590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.508619 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.508691 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:00:24.508883 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.509454 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.510461 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.510877 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.510967 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.511047 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.511156 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.511201 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 07:00:24.511250 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.508378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:34517 (local address 127.1.74.67:33767)
0926 07:00:24.508433 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:24.508446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:24.508492 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:24.508711 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:24.508713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:24.508715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:00:24.508790 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.508793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.509545 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.509546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.510365 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.510367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.510981 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.510981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.511074 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.511077 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:00:24.511135 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:00:24.511173 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":31}
I20260926 07:00:24.511242 17664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.508384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:41377 (local address 127.1.74.67:45947)
0926 07:00:24.508603 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:24.508612 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:24.508660 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:24.508815 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:24.508816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:24.508818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:00:24.508907 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.508909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.509544 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.509545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.510377 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:24.510379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.510954 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.510955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:24.511018 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.511023 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:00:24.511078 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 07:00:24.511144 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":120,"thread_start_us":89,"threads_started":1}
I20260926 07:00:24.511572 17315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de8ef40db8d43e390c53782427b4775" candidate_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f696558ad2452da61883f6fcc428ea" is_pre_election: true
I20260926 07:00:24.511801 17315 raft_consensus.cc:2454] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587d36d0f2cb4ac7a1cb62dc1c8b45ec in term 0.
I20260926 07:00:24.511853 17471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de8ef40db8d43e390c53782427b4775" candidate_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c867f7fbb9495f943b9a4a95cde8db" is_pre_election: true
I20260926 07:00:24.511996 17471 raft_consensus.cc:2454] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 587d36d0f2cb4ac7a1cb62dc1c8b45ec in term 0.
I20260926 07:00:24.512054 17536 leader_election.cc:302] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f696558ad2452da61883f6fcc428ea, 587d36d0f2cb4ac7a1cb62dc1c8b45ec; no voters:
I20260926 07:00:24.512229 17662 raft_consensus.cc:2781] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:00:24.512295 17662 raft_consensus.cc:482] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:00:24.512313 17662 raft_consensus.cc:3037] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:00:24.513101 17662 raft_consensus.cc:504] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.513243 17662 leader_election.cc:288] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [CANDIDATE]: Term 1 election: Requested vote from peers 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377), 20f696558ad2452da61883f6fcc428ea (127.1.74.65:34517)
I20260926 07:00:24.513391 17471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de8ef40db8d43e390c53782427b4775" candidate_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c867f7fbb9495f943b9a4a95cde8db"
I20260926 07:00:24.513405 17315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de8ef40db8d43e390c53782427b4775" candidate_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20f696558ad2452da61883f6fcc428ea"
I20260926 07:00:24.513481 17315 raft_consensus.cc:3037] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:00:24.513482 17471 raft_consensus.cc:3037] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:00:24.514317 17471 raft_consensus.cc:2454] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587d36d0f2cb4ac7a1cb62dc1c8b45ec in term 1.
I20260926 07:00:24.514333 17315 raft_consensus.cc:2454] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 587d36d0f2cb4ac7a1cb62dc1c8b45ec in term 1.
I20260926 07:00:24.514483 17536 leader_election.cc:302] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f696558ad2452da61883f6fcc428ea, 587d36d0f2cb4ac7a1cb62dc1c8b45ec; no voters:
I20260926 07:00:24.514596 17662 raft_consensus.cc:2781] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:00:24.514735 17662 raft_consensus.cc:686] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 1 LEADER]: Becoming Leader. State: Replica: 587d36d0f2cb4ac7a1cb62dc1c8b45ec, State: Running, Role: LEADER
I20260926 07:00:24.514830 17662 consensus_queue.cc:237] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:24.515699 17190 catalog_manager.cc:5495] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67). New cstate: current_term: 1 leader_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } health_report { overall_health: UNKNOWN } } }
W20260926 07:00:24.528432 17380 tablet.cc:2369] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:00:24.536079 17650 tablet.cc:2369] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:00:24.545744 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.543332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.1.74.67:37997)
0926 07:00:24.543386 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:24.543389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:24.543398 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:00:24.543436 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:24.543437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:24.543467 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:24.543550 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:00:24.543691 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.544161 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.544912 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:24.545127 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:24.545212 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:24.545390 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:00:24.545502 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:00:24.545589 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:24.545612 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 07:00:24.545634 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 07:00:24.547583 17315 raft_consensus.cc:1261] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Refusing update from remote peer 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:00:24.547600 17471 raft_consensus.cc:1261] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [term 1 FOLLOWER]: Refusing update from remote peer 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:00:24.547976 17662 consensus_queue.cc:1035] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:00:24.548131 17665 consensus_queue.cc:1035] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:00:24.550009 17668 mvcc.cc:205] Tried to move back new op lower bound from 7333503076543582208 to 7333503076413419520. Current Snapshot: MvccSnapshot[applied={T|T < 7333503076543582208}]
I20260926 07:00:24.550272 17669 mvcc.cc:205] Tried to move back new op lower bound from 7333503076543582208 to 7333503076413419520. Current Snapshot: MvccSnapshot[applied={T|T < 7333503076543582208}]
I20260926 07:00:24.552801 17190 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:00:24.552904 17190 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:00:24.553669 17190 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Sending DeleteTablet for 3 replicas of tablet 1de8ef40db8d43e390c53782427b4775
I20260926 07:00:24.553936 17298 tablet_service.cc:1514] Processing DeleteTablet for tablet 1de8ef40db8d43e390c53782427b4775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:00:24 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:59444
I20260926 07:00:24.554031 17388 tablet_replica.cc:330] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: stopping tablet replica
I20260926 07:00:24.554003 17583 tablet_service.cc:1514] Processing DeleteTablet for tablet 1de8ef40db8d43e390c53782427b4775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:00:24 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:40894
I20260926 07:00:24.554010 17451 tablet_service.cc:1514] Processing DeleteTablet for tablet 1de8ef40db8d43e390c53782427b4775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:00:24 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:60140
I20260926 07:00:24.554086 17388 raft_consensus.cc:2229] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:00:24.554117 17388 raft_consensus.cc:2258] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:00:24.554256 17677 tablet_replica.cc:330] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: stopping tablet replica
I20260926 07:00:24.554306 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 17229
I20260926 07:00:24.554363 17677 raft_consensus.cc:2229] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:00:24.554415 17388 ts_tablet_manager.cc:1905] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:00:24.554455 17677 raft_consensus.cc:2258] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:00:24.554591 17678 tablet_replica.cc:330] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: stopping tablet replica
I20260926 07:00:24.554672 17678 raft_consensus.cc:2229] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:00:24.554757 17678 raft_consensus.cc:2258] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:00:24.555172 17678 ts_tablet_manager.cc:1905] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:00:24.557286 17677 ts_tablet_manager.cc:1905] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:00:24.559105 17677 ts_tablet_manager.cc:1918] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:00:24.559180 17677 log.cc:1196] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/1de8ef40db8d43e390c53782427b4775
I20260926 07:00:24.559376 17678 ts_tablet_manager.cc:1918] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:00:24.559485 17677 ts_tablet_manager.cc:1939] T 1de8ef40db8d43e390c53782427b4775 P 587d36d0f2cb4ac7a1cb62dc1c8b45ec: Deleting consensus metadata
I20260926 07:00:24.559510 17678 log.cc:1196] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/1de8ef40db8d43e390c53782427b4775
I20260926 07:00:24.559881 17177 catalog_manager.cc:4841] TS 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997): tablet 1de8ef40db8d43e390c53782427b4775 (table test-table [id=67f4c4b2fc3746f1958925f1361d8e78]) successfully deleted
I20260926 07:00:24.559919 17678 ts_tablet_manager.cc:1939] T 1de8ef40db8d43e390c53782427b4775 P 18c867f7fbb9495f943b9a4a95cde8db: Deleting consensus metadata
I20260926 07:00:24.560397 17177 catalog_manager.cc:4841] TS 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377): tablet 1de8ef40db8d43e390c53782427b4775 (table test-table [id=67f4c4b2fc3746f1958925f1361d8e78]) successfully deleted
W20260926 07:00:24.561770 17175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:34517 (error 108)
I20260926 07:00:24.561996 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 17384
I20260926 07:00:24.562263 17654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:24.562154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:34517 (local address 127.0.0.1:44372)
0926 07:00:24.562203 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:24.562233 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:34517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 07:00:24.563030 17175 catalog_manager.cc:4584] TS 20f696558ad2452da61883f6fcc428ea (127.1.74.65:34517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1de8ef40db8d43e390c53782427b4775: Network error: Client connection negotiation failed: client connection to 127.1.74.65:34517: connect: Connection refused (error 111)
I20260926 07:00:24.566458 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 17520
I20260926 07:00:24.570940 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 17158
I20260926 07:00:44.576437 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42423
--webserver_interface=127.1.74.126
--webserver_port=45939
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:00:44.658644 17680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:44.658882 17680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:44.658915 17680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:44.660046 17680 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:00:44.660109 17680 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:00:44.660130 17680 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:00:44.660148 17680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:00:44.660180 17680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:44.660210 17680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:44.660234 17680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:00:44.660259 17680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:00:44.661801 17680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=45939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:44.662117 17680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:44.662384 17680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:44.667832 17685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:44.667896 17680 server_base.cc:1034] running on GCE node
W20260926 07:00:44.667817 17688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:44.667820 17686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:44.668326 17680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:44.668684 17680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:44.669862 17680 hybrid_clock.cc:648] HybridClock initialized: now 1790406044669853 us; error 49 us; skew 500 ppm
Sep 26 07:00:44 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:44.676959 17680 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:00:44.677414 17680 webserver.cc:460] Webserver started at http://127.1.74.126:45939/ using document root <none> and password file <none>
I20260926 07:00:44.677666 17680 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:44.677734 17680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:44.679193 17680 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:44.679988 17695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:44.680198 17680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:44.680306 17680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "3de3469878b44214939f5a73bda745ac"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "a1e617bf76008ffe26b5782b0236c937"
server_key_iv: "6b7cd3e259eff9d61fa78cdab08d424f"
server_key_version: "encryptionkey@0"
I20260926 07:00:44.680732 17680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:44.688211 17680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:44.690215 17680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:44.690402 17680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:44.694824 17680 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:42423
I20260926 07:00:44.694854 17747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:42423 every 8 connection(s)
I20260926 07:00:44.695339 17680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:00:44.696811 17748 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:00:44.697414 17748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Bootstrap starting.
I20260926 07:00:44.701656 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 17680
I20260926 07:00:44.701947 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:34517
--local_ip_for_outbound_sockets=127.1.74.65
--tserver_master_addrs=127.1.74.126:42423
--webserver_port=39365
--webserver_interface=127.1.74.65
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:00:44.706010 17748 log.cc:824] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:44.708324 17748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:00:44.708638 17748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Bootstrap complete.
I20260926 07:00:44.710795 17748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:44.711017 17748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de3469878b44214939f5a73bda745ac, State: Initialized, Role: FOLLOWER
I20260926 07:00:44.711218 17748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:44.711329 17748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:00:44.711385 17748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:00:44.711437 17748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:00:44.712427 17748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:44.712548 17748 leader_election.cc:302] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de3469878b44214939f5a73bda745ac; no voters:
I20260926 07:00:44.712779 17748 leader_election.cc:288] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:00:44.712821 17752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:00:44.712937 17752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [term 3 LEADER]: Becoming Leader. State: Replica: 3de3469878b44214939f5a73bda745ac, State: Running, Role: LEADER
I20260926 07:00:44.713061 17748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:00:44.713088 17752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } }
I20260926 07:00:44.713449 17753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3de3469878b44214939f5a73bda745ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } } }
I20260926 07:00:44.713439 17754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de3469878b44214939f5a73bda745ac. Latest consensus state: current_term: 3 leader_uuid: "3de3469878b44214939f5a73bda745ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de3469878b44214939f5a73bda745ac" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42423 } } }
I20260926 07:00:44.713601 17753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: This master's current role is: LEADER
I20260926 07:00:44.713608 17754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac [sys.catalog]: This master's current role is: LEADER
I20260926 07:00:44.714027 17760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:00:44.714941 17760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:00:44.715042 17760 catalog_manager.cc:1259] Loaded cluster ID: 0183602ff89f430783bb522bb60c8723
I20260926 07:00:44.715118 17760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:00:44.716100 17760 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:00:44.716302 17760 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Loaded TSK: 0
I20260926 07:00:44.716360 17760 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3de3469878b44214939f5a73bda745ac: Loaded TSK: 1
F20260926 07:00:44.716605 17760 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc73b8f59ad google::LogMessage::Fail() at ??:0
@ 0x7fc73b8f888a google::LogMessage::SendToLog() at ??:0
@ 0x7fc73b8f54de google::LogMessage::Flush() at ??:0
@ 0x7fc73b8f6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc73e062104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc73e05fd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc74384f1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc74384f48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc74384f594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc743815c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc74381fad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc73d7c30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc73d7b8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc73e61c6db start_thread at ??:0
@ 0x7fc73a3f361f clone at ??:0
W20260926 07:00:44.802698 17750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:44.802943 17750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:44.802992 17750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:44.804261 17750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:44.804333 17750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:44.804380 17750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 07:00:44.806034 17750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:34517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=39365
--tserver_master_addrs=127.1.74.126:42423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:44.806314 17750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:44.806582 17750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:44.812793 17790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:44.812811 17789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:44.812785 17792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:44.812979 17750 server_base.cc:1034] running on GCE node
I20260926 07:00:44.813304 17750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:44.813584 17750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:44.814740 17750 hybrid_clock.cc:648] HybridClock initialized: now 1790406044814719 us; error 36 us; skew 500 ppm
Sep 26 07:00:44 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:44.817329 17750 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 07:00:44.817834 17750 webserver.cc:460] Webserver started at http://127.1.74.65:39365/ using document root <none> and password file <none>
I20260926 07:00:44.818084 17750 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:44.818153 17750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:44.819586 17750 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:44.820387 17799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:44.820632 17750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:44.820730 17750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "20f696558ad2452da61883f6fcc428ea"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "d21d6aa5f40c90bd83f8d4048b03c5df"
server_key_iv: "f252ca5c106c53ee30ab9fa8fbbc8d56"
server_key_version: "encryptionkey@0"
I20260926 07:00:44.821197 17750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:44.836158 17750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:44.838851 17750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:44.839043 17750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:44.839365 17750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:00:44.839879 17806 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:00:44.841431 17750 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:00:44.841496 17750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 07:00:44.841533 17750 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:00:44.842505 17750 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:00:44.842556 17750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 07:00:44.842639 17806 tablet_bootstrap.cc:492] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Bootstrap starting.
I20260926 07:00:44.844007 17806 log.cc:824] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Log is configured to *not* fsync() on all Append() calls
I20260926 07:00:44.847143 17806 tablet_bootstrap.cc:492] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:00:44.847512 17806 tablet_bootstrap.cc:492] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Bootstrap complete.
I20260926 07:00:44.847666 17806 ts_tablet_manager.cc:1397] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20260926 07:00:44.849572 17806 raft_consensus.cc:348] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:44.849876 17806 raft_consensus.cc:729] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f696558ad2452da61883f6fcc428ea, State: Initialized, Role: FOLLOWER
I20260926 07:00:44.850045 17806 consensus_queue.cc:260] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:44.850244 17750 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:34517
I20260926 07:00:44.850333 17806 ts_tablet_manager.cc:1428] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 07:00:44.850626 17750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 07:00:44.850811 17915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:34517 every 8 connection(s)
I20260926 07:00:44.857406 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 17750
I20260926 07:00:44.857729 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:41377
--local_ip_for_outbound_sockets=127.1.74.66
--tserver_master_addrs=127.1.74.126:42423
--webserver_port=34613
--webserver_interface=127.1.74.66
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:00:44.935245 17918 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:00:44.851330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:34411)
0926 07:00:44.851550 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:44.851592 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:44.851725 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:44.935025 (+ 83300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":118,"threads_started":1}
W20260926 07:00:44.935568 17916 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:42423 (0 consecutive failures): Network error: Failed to ping master at 127.1.74.126:42423: Client connection negotiation failed: client connection to 127.1.74.126:42423: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:00:44.935954 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:44.935819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:51979)
0926 07:00:44.935881 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:44.935927 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 07:00:44.936295 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:44.936202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:44091)
0926 07:00:44.936246 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:44.936273 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 07:00:44.936417 17916 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:00:44.946033 17919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:44.946264 17919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:44.946290 17919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:44.947511 17919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:44.947557 17919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:44.947585 17919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 07:00:44.949025 17919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:41377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=34613
--tserver_master_addrs=127.1.74.126:42423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:44.949259 17919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:44.949540 17919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:44.955406 17925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:44.955395 17924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:44.955395 17927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:44.955775 17919 server_base.cc:1034] running on GCE node
I20260926 07:00:44.956024 17919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:44.956272 17919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:44.957446 17919 hybrid_clock.cc:648] HybridClock initialized: now 1790406044957443 us; error 40 us; skew 500 ppm
Sep 26 07:00:44 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:44.960192 17919 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 07:00:44.960765 17919 webserver.cc:460] Webserver started at http://127.1.74.66:34613/ using document root <none> and password file <none>
I20260926 07:00:44.961015 17919 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:44.961110 17919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:44.962726 17919 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:00:44.963487 17934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:44.963668 17919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:00:44.963783 17919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "18c867f7fbb9495f943b9a4a95cde8db"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "9716c9dcb511afd01cf0c5b76054b576"
server_key_iv: "b77b031ce83fad681f15379a2cf29f61"
server_key_version: "encryptionkey@0"
I20260926 07:00:44.964184 17919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:44.977169 17919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:44.979991 17919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:44.980207 17919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:44.980496 17919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:00:44.980851 17919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:00:44.980911 17919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:44.980950 17919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:00:44.981004 17919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:44.987823 17919 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:41377
I20260926 07:00:44.987877 18047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:41377 every 8 connection(s)
I20260926 07:00:44.988270 17919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 07:00:44.989416 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:44.988972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:52895)
0926 07:00:44.989175 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:44.989257 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
W20260926 07:00:44.989874 18048 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:42423 (0 consecutive failures): Network error: Failed to ping master at 127.1.74.126:42423: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
I20260926 07:00:44.990196 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:44.990089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:45009)
0926 07:00:44.990148 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:44.990177 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 07:00:44.990459 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:44.990405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:35325)
0926 07:00:44.990427 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:44.990446 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 07:00:44.990590 18048 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:00:44.993422 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 17919
I20260926 07:00:44.993714 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:37997
--local_ip_for_outbound_sockets=127.1.74.67
--tserver_master_addrs=127.1.74.126:42423
--webserver_port=38855
--webserver_interface=127.1.74.67
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:00:45.077970 18051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:00:45.078186 18051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:00:45.078213 18051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:00:45.079907 18051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:00:45.079968 18051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:00:45.080013 18051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 07:00:45.081995 18051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:37997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=38855
--tserver_master_addrs=127.1.74.126:42423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:00:45.082279 18051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:00:45.082538 18051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:00:45.087630 18057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:45.087641 18059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:00:45.087641 18056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:00:45.087937 18051 server_base.cc:1034] running on GCE node
I20260926 07:00:45.088164 18051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:00:45.088376 18051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:00:45.089519 18051 hybrid_clock.cc:648] HybridClock initialized: now 1790406045089505 us; error 29 us; skew 500 ppm
Sep 26 07:00:45 dist-test-slave-sfvt krb5kdc[16612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406045, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:00:45.092108 18051 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 07:00:45.092607 18051 webserver.cc:460] Webserver started at http://127.1.74.67:38855/ using document root <none> and password file <none>
I20260926 07:00:45.092854 18051 fs_manager.cc:359] Metadata directory not provided
I20260926 07:00:45.092900 18051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:00:45.094275 18051 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:00:45.094893 18066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:45.095072 18051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:00:45.095165 18051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec"
format_stamp: "Formatted at 2026-09-26 07:00:03 on dist-test-slave-sfvt"
server_key: "cdd9d81a566474d3f4204e3dd7484ed3"
server_key_iv: "5d5f8753eb7241638aa11b9e8c34c818"
server_key_version: "encryptionkey@0"
I20260926 07:00:45.095603 18051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:00:45.114888 18051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:00:45.117288 18051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:00:45.117483 18051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:00:45.117754 18051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:00:45.118076 18051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:00:45.118140 18051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:45.118185 18051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:00:45.118211 18051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:00:45.124642 18051 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:37997
I20260926 07:00:45.124689 18179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:37997 every 8 connection(s)
I20260926 07:00:45.125097 18051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 07:00:45.126312 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:45.125927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:45011)
0926 07:00:45.126081 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:45.126168 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
W20260926 07:00:45.126714 18180 heartbeater.cc:644] Failed to heartbeat to 127.1.74.126:42423 (0 consecutive failures): Network error: Failed to ping master at 127.1.74.126:42423: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
I20260926 07:00:45.127002 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:45.126925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:38645)
0926 07:00:45.126959 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:45.126986 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 07:00:45.127315 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:45.127218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:37143)
0926 07:00:45.127240 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:45.127293 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 07:00:45.127424 18180 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:00:45.129249 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 18051
I20260926 07:00:45.937250 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:45.936906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:40663)
0926 07:00:45.937060 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:45.937167 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260926 07:00:45.991086 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:45.990900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:55857)
0926 07:00:45.991008 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:45.991037 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
I20260926 07:00:46.043390 18186 raft_consensus.cc:482] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:00:46.043576 18186 raft_consensus.cc:504] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:46.044009 18186 leader_election.cc:288] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997), 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377)
Sep 26 07:00:46 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.66@KRBTEST.COM
Sep 26 07:00:46 dist-test-slave-sfvt krb5kdc[16612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.67@KRBTEST.COM
I20260926 07:00:46.056124 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:46.044267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:35163 (local address 127.1.74.67:37997)
0926 07:00:46.044476 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:46.044488 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:46.044518 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:00:46.045124 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:46.045127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:46.045204 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:46.045348 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:46.046159 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:46.047527 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:46.048583 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:46.048983 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:46.049108 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:46.051941 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:46.051945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:46.051945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:46.051958 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:46.053861 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:46.054554 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:46.054557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:46.054558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:46.054689 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:46.055046 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:46.055047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:46.055048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:46.055420 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:46.055508 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:46.055874 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 07:00:46.056016 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
I20260926 07:00:46.056136 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:46.044395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:42611 (local address 127.1.74.66:41377)
0926 07:00:46.044533 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 07:00:46.044587 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0926 07:00:46.044615 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:00:46.045140 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:00:46.045143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:00:46.045192 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:00:46.045348 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:00:46.046400 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:46.047381 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:46.048561 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:00:46.048978 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:00:46.049097 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:46.051727 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:00:46.051732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:00:46.051732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:00:46.051744 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:00:46.053869 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:46.054554 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:46.054557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:46.054558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:46.054697 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:00:46.055047 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:00:46.055047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:00:46.055048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:00:46.055420 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:00:46.055508 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:00:46.055874 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 07:00:46.056016 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":52,"threads_started":1}
I20260926 07:00:46.056156 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:46.044141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:37997 (local address 127.1.74.65:35163)
0926 07:00:46.044208 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:46.044224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:46.044298 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:46.045420 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:46.045423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:46.045771 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:00:46.046017 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:46.046023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:46.047682 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:46.047683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:46.048457 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:46.048459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:46.049089 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:46.049090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:46.049211 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:46.049877 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:00:46.049881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:00:46.051831 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:00:46.053980 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:46.053981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:46.053981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:46.054461 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:46.054769 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:46.054770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:46.054770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:46.054971 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:46.055498 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:00:46.055505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:00:46.055674 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 07:00:46.055798 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:00:46.056021 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":49}
I20260926 07:00:46.056180 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:46.044295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:41377 (local address 127.1.74.65:42611)
0926 07:00:46.044408 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:46.044418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:00:46.044469 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:00:46.045472 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:00:46.045473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:00:46.045770 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:00:46.046257 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:46.046260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:46.047502 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:46.047503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:46.048458 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:00:46.048460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:00:46.049072 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:00:46.049073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:00:46.049212 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:00:46.049811 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:00:46.049819 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:00:46.051592 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:00:46.053959 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:46.053969 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:46.053972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:46.054461 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:46.054769 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:00:46.054770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:00:46.054770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:00:46.054971 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:00:46.055498 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:00:46.055505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:00:46.055674 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 07:00:46.055798 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:00:46.056075 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":183,"thread_start_us":73,"threads_started":1}
W20260926 07:00:46.057101 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
W20260926 07:00:46.057196 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
I20260926 07:00:46.057226 17802 leader_election.cc:302] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20f696558ad2452da61883f6fcc428ea; no voters: 18c867f7fbb9495f943b9a4a95cde8db, 587d36d0f2cb4ac7a1cb62dc1c8b45ec
I20260926 07:00:46.057323 18186 raft_consensus.cc:2726] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:00:46.128017 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:46.127808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:51179)
0926 07:00:46.127938 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:46.127977 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260926 07:00:46.938047 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:46.937788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:55145)
0926 07:00:46.937956 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:46.938001 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260926 07:00:46.991928 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:46.991715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:36293)
0926 07:00:46.991845 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:46.991885 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260926 07:00:47.128696 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:47.128465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:36823)
0926 07:00:47.128618 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:47.128657 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260926 07:00:47.582231 18194 raft_consensus.cc:482] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:00:47.582366 18194 raft_consensus.cc:504] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:47.582584 18194 leader_election.cc:288] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997), 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377)
W20260926 07:00:47.582991 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
W20260926 07:00:47.583060 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
I20260926 07:00:47.583074 17802 leader_election.cc:302] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20f696558ad2452da61883f6fcc428ea; no voters: 18c867f7fbb9495f943b9a4a95cde8db, 587d36d0f2cb4ac7a1cb62dc1c8b45ec
I20260926 07:00:47.583140 18194 raft_consensus.cc:2726] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:00:47.938714 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:47.938513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:57529)
0926 07:00:47.938632 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:47.938668 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260926 07:00:47.992628 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:47.992387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:56113)
0926 07:00:47.992530 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:47.992572 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260926 07:00:48.129503 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:48.129288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:48327)
0926 07:00:48.129418 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:48.129458 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260926 07:00:48.939458 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:48.939163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:52107)
0926 07:00:48.939373 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:48.939411 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260926 07:00:48.993320 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:48.993109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:57683)
0926 07:00:48.993238 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:48.993277 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260926 07:00:49.130234 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:49.129986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:45307)
0926 07:00:49.130150 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:49.130190 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260926 07:00:49.858127 18201 raft_consensus.cc:482] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:00:49.858237 18201 raft_consensus.cc:504] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:49.858424 18201 leader_election.cc:288] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997), 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377)
W20260926 07:00:49.858845 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
W20260926 07:00:49.858911 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
I20260926 07:00:49.858927 17802 leader_election.cc:302] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20f696558ad2452da61883f6fcc428ea; no voters: 18c867f7fbb9495f943b9a4a95cde8db, 587d36d0f2cb4ac7a1cb62dc1c8b45ec
I20260926 07:00:49.859045 18201 raft_consensus.cc:2726] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:00:49.940132 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:49.939899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:40705)
0926 07:00:49.940042 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:49.940079 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260926 07:00:49.994050 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:49.993832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:37479)
0926 07:00:49.993970 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:49.994007 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260926 07:00:50.130831 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:50.130640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:42823)
0926 07:00:50.130759 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:50.130793 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260926 07:00:50.940841 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:50.940606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:43223)
0926 07:00:50.940742 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:50.940780 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260926 07:00:50.994772 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:50.994530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:33001)
0926 07:00:50.994674 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:50.994719 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260926 07:00:51.131655 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:51.131442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:39507)
0926 07:00:51.131571 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:51.131616 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260926 07:00:51.916761 18208 raft_consensus.cc:482] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:00:51.916903 18208 raft_consensus.cc:504] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:51.917117 18208 leader_election.cc:288] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997), 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377)
W20260926 07:00:51.917557 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
W20260926 07:00:51.917613 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
I20260926 07:00:51.917629 17802 leader_election.cc:302] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20f696558ad2452da61883f6fcc428ea; no voters: 18c867f7fbb9495f943b9a4a95cde8db, 587d36d0f2cb4ac7a1cb62dc1c8b45ec
I20260926 07:00:51.917743 18208 raft_consensus.cc:2726] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:00:51.941533 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:51.941262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:51569)
0926 07:00:51.941437 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:51.941488 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":77,"threads_started":1}
I20260926 07:00:51.995610 18210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:51.995364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:45721)
0926 07:00:51.995522 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:51.995553 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260926 07:00:52.132308 18211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:52.132083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:56449)
0926 07:00:52.132202 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:52.132234 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260926 07:00:52.942169 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:52.941969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:41495)
0926 07:00:52.942096 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:52.942123 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260926 07:00:52.996300 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:52.996097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:60745)
0926 07:00:52.996219 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:52.996255 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260926 07:00:53.132942 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:53.132720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:51209)
0926 07:00:53.132849 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:53.132886 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260926 07:00:53.942812 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:53.942572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:49195)
0926 07:00:53.942726 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:53.942765 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260926 07:00:53.996937 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:53.996735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:39947)
0926 07:00:53.996856 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:53.996891 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260926 07:00:54.133623 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:54.133430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:55049)
0926 07:00:54.133545 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:54.133581 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260926 07:00:54.943640 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:54.943404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:34071)
0926 07:00:54.943552 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:54.943592 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260926 07:00:54.997597 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:54.997372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:58563)
0926 07:00:54.997516 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:54.997553 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260926 07:00:55.134299 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:55.134058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:45559)
0926 07:00:55.134177 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:55.134216 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260926 07:00:55.944352 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:55.944131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:52689)
0926 07:00:55.944264 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:55.944304 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260926 07:00:55.964352 18222 raft_consensus.cc:482] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:00:55.964440 18222 raft_consensus.cc:504] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587d36d0f2cb4ac7a1cb62dc1c8b45ec" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 37997 } } peers { permanent_uuid: "18c867f7fbb9495f943b9a4a95cde8db" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 41377 } } peers { permanent_uuid: "20f696558ad2452da61883f6fcc428ea" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 34517 } }
I20260926 07:00:55.964635 18222 leader_election.cc:288] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997), 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377)
W20260926 07:00:55.965046 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 587d36d0f2cb4ac7a1cb62dc1c8b45ec (127.1.74.67:37997): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
W20260926 07:00:55.965130 17802 leader_election.cc:341] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 18c867f7fbb9495f943b9a4a95cde8db (127.1.74.66:41377): Not found: Tablet not found: 1de8ef40db8d43e390c53782427b4775
I20260926 07:00:55.965162 17802 leader_election.cc:302] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20f696558ad2452da61883f6fcc428ea; no voters: 18c867f7fbb9495f943b9a4a95cde8db, 587d36d0f2cb4ac7a1cb62dc1c8b45ec
I20260926 07:00:55.965298 18222 raft_consensus.cc:2726] T 1de8ef40db8d43e390c53782427b4775 P 20f696558ad2452da61883f6fcc428ea [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:00:55.998463 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:55.998184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:55329)
0926 07:00:55.998361 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:55.998422 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260926 07:00:56.135035 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:56.134810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:52749)
0926 07:00:56.134951 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:56.134988 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260926 07:00:56.945159 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:56.944943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:36603)
0926 07:00:56.945085 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:56.945115 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260926 07:00:56.999239 18226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:56.998972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:54683)
0926 07:00:56.999120 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:56.999153 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260926 07:00:57.135756 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:57.135545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:42789)
0926 07:00:57.135672 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:57.135710 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260926 07:00:57.945921 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:57.945672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:55853)
0926 07:00:57.945834 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:57.945874 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260926 07:00:58.000020 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:57.999757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:37629)
0926 07:00:57.999915 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:57.999957 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260926 07:00:58.136463 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:58.136226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:33035)
0926 07:00:58.136372 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:58.136414 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260926 07:00:58.946766 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:58.946546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:48225)
0926 07:00:58.946678 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:58.946719 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260926 07:00:59.000718 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:59.000486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:58161)
0926 07:00:59.000635 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:59.000672 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260926 07:00:59.137112 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:59.136928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:42229)
0926 07:00:59.137038 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:59.137067 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
I20260926 07:00:59.947698 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:00:59.947435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.65:43011)
0926 07:00:59.947612 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:00:59.947652 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260926 07:01:00.001348 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:00.001144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.66:37635)
0926 07:01:00.001269 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:00.001302 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260926 07:01:00.137526 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 17750
I20260926 07:01:00.143635 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 17919
I20260926 07:01:00.143700 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:00.137646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42423 (local address 127.1.74.67:32857)
0926 07:01:00.143595 (+ 5949us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:00.143641 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.126:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5922,"thread_start_us":56,"threads_started":1}
I20260926 07:01:00.149423 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 18051
2026-09-26T07:01:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58334 ms)
[----------] 30 tests from SecurityITest (226650 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:00 dist-test-slave-sfvt krb5kdc[18239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:00 dist-test-slave-sfvt krb5kdc[18239](info): set up 2 sockets
Sep 26 07:01:00 dist-test-slave-sfvt krb5kdc[18239](info): commencing operation
krb5kdc: starting...
W20260926 07:01:01.206939 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:01 dist-test-slave-sfvt krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:01.326490 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36359
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:36359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:01.407892 18252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:01.408126 18252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:01.408162 18252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:01.409298 18252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:01.409363 18252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:01.409386 18252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:01.409417 18252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:01.409444 18252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:01.410884 18252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:36359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:01.411185 18252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:01.411439 18252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:01:01.416750 18252 server_base.cc:1034] running on GCE node
W20260926 07:01:01.416728 18258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:01.416744 18257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:01.416734 18260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:01.417266 18252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:01.417610 18252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:01.418774 18252 hybrid_clock.cc:648] HybridClock initialized: now 1790406061418753 us; error 34 us; skew 500 ppm
Sep 26 07:01:01 dist-test-slave-sfvt krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:01.421345 18252 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:01.421813 18252 webserver.cc:460] Webserver started at http://127.1.74.126:36979/ using document root <none> and password file <none>
I20260926 07:01:01.422035 18252 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:01.422101 18252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:01.422238 18252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:01.423364 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "12c4d28b60a9479c983195cfbe77d499"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "fb0adb7c942c19606e6138b0f973edaa"
server_key_iv: "97e9e653c47841b3d16f45cf0ea3cd3d"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.423724 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "12c4d28b60a9479c983195cfbe77d499"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "fb0adb7c942c19606e6138b0f973edaa"
server_key_iv: "97e9e653c47841b3d16f45cf0ea3cd3d"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.424994 18252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:01.425694 18267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.425855 18252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:01.425952 18252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "12c4d28b60a9479c983195cfbe77d499"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "fb0adb7c942c19606e6138b0f973edaa"
server_key_iv: "97e9e653c47841b3d16f45cf0ea3cd3d"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.426033 18252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:01.436594 18252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:01.438948 18252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:01.439109 18252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:01.443367 18252 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:36359
I20260926 07:01:01.443454 18319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:36359 every 8 connection(s)
I20260926 07:01:01.443826 18252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:01.444698 18320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:01.447031 18320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499: Bootstrap starting.
I20260926 07:01:01.447677 18320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:01.447975 18320 log.cc:824] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:01.448670 18320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499: No bootstrap required, opened a new log
I20260926 07:01:01.450259 18320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4d28b60a9479c983195cfbe77d499" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36359 } }
I20260926 07:01:01.450412 18320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:01.450474 18320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c4d28b60a9479c983195cfbe77d499, State: Initialized, Role: FOLLOWER
I20260926 07:01:01.450620 18320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4d28b60a9479c983195cfbe77d499" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36359 } }
I20260926 07:01:01.450680 18320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:01.450745 18320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:01.450794 18320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:01.451021 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 18252
I20260926 07:01:01.451136 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:01.451362 1321 external_mini_cluster.cc:1428] Setting key d120f156be06334a444b129ad359c780
I20260926 07:01:01.451505 18320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4d28b60a9479c983195cfbe77d499" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36359 } }
I20260926 07:01:01.451630 18320 leader_election.cc:302] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12c4d28b60a9479c983195cfbe77d499; no voters:
I20260926 07:01:01.451805 18320 leader_election.cc:288] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:01.452137 18320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:01.451824 18323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:01.453828 18323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [term 1 LEADER]: Becoming Leader. State: Replica: 12c4d28b60a9479c983195cfbe77d499, State: Running, Role: LEADER
I20260926 07:01:01.453939 18323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4d28b60a9479c983195cfbe77d499" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36359 } }
I20260926 07:01:01.454252 18323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c4d28b60a9479c983195cfbe77d499. Latest consensus state: current_term: 1 leader_uuid: "12c4d28b60a9479c983195cfbe77d499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4d28b60a9479c983195cfbe77d499" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36359 } } }
I20260926 07:01:01.454329 18323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:01.454500 18337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12c4d28b60a9479c983195cfbe77d499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4d28b60a9479c983195cfbe77d499" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36359 } } }
I20260926 07:01:01.454761 18337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499 [sys.catalog]: This master's current role is: LEADER
W20260926 07:01:01.455081 18340 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:01.455144 18340 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:01.455331 18339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:01.455490 18339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:01.457067 18339 catalog_manager.cc:1347] Generated new cluster ID: 69f65642ff644e63875922aff0ed96bc
I20260926 07:01:01.457120 18339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:01 dist-test-slave-sfvt krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:01.462097 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.451858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.1.74.126:36359)
0926 07:01:01.452017 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:01.452026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:01.452063 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:01:01.452167 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:01.452172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:01.452228 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:01.452383 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:01.453253 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.454501 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.455643 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.456047 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.456165 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.458710 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:01.458714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:01.458715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:01.458728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:01.460515 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.460893 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.460897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.460897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.461008 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.461235 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.461236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.461237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.461475 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:01.461646 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:01.461730 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 07:01:01.461884 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260926 07:01:01.462525 18339 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:01.463208 18339 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:01.474237 18339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499: Generated new TSK 0
I20260926 07:01:01.474445 18339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 07:01:01.506352 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36359
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:01.586983 18346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:01.587189 18346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:01.587212 18346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:01.588369 18346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:01.588410 18346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:01.588436 18346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 07:01:01.589838 18346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:01.590072 18346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:01.590319 18346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:01.595373 18352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:01.595403 18354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:01.595409 18351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:01.595731 18346 server_base.cc:1034] running on GCE node
I20260926 07:01:01.595950 18346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:01.596199 18346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:01.597361 18346 hybrid_clock.cc:648] HybridClock initialized: now 1790406061597356 us; error 35 us; skew 500 ppm
Sep 26 07:01:01 dist-test-slave-sfvt krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:01.599817 18346 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 07:01:01.600219 18346 webserver.cc:460] Webserver started at http://127.1.74.65:34135/ using document root <none> and password file <none>
I20260926 07:01:01.600435 18346 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:01.600499 18346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:01.600659 18346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:01.601672 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "1e21546c4ef7475680811f549419645e"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "ce27170024aa21741cf5355f7d024c62"
server_key_iv: "cb4eb5bb0959431d9deb7d3b250057b3"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.602031 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "1e21546c4ef7475680811f549419645e"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "ce27170024aa21741cf5355f7d024c62"
server_key_iv: "cb4eb5bb0959431d9deb7d3b250057b3"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.603336 18346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:01.604071 18361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.604240 18346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:01.604336 18346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "1e21546c4ef7475680811f549419645e"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "ce27170024aa21741cf5355f7d024c62"
server_key_iv: "cb4eb5bb0959431d9deb7d3b250057b3"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.604419 18346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:01.621642 18346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:01.623569 18346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:01.623723 18346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:01.623965 18346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:01.624249 18346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:01.624297 18346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.624342 18346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:01.624366 18346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.630271 18346 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:35567
I20260926 07:01:01.630373 18474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:35567 every 8 connection(s)
I20260926 07:01:01.630659 18346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 07:01:01.631174 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 18346
I20260926 07:01:01.631311 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:01.631510 1321 external_mini_cluster.cc:1428] Setting key e40d3d2a0e800b5e36df1f7557286648
Sep 26 07:01:01 dist-test-slave-sfvt krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
I20260926 07:01:01.641734 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.631364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:50871 (local address 127.1.74.126:36359)
0926 07:01:01.631417 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:01.631419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:01.631655 (+ 236us) server_negotiation.cc:407] Connection header received
0926 07:01:01.632350 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:01.632351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:01.632367 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:01.632441 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:01.633395 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.633943 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.635625 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.635866 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.635930 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.638468 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:01.638471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:01.638471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:01.638479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:01.640155 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.640610 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.640611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.640612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.640663 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.640919 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.640920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.640920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.641033 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:01.641150 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:01.641599 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 07:01:01.641668 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:01:01.641999 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.631365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36359 (local address 127.1.74.65:50871)
0926 07:01:01.631535 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:01.631577 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:01.632238 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:01.632492 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:01.632496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:01.633057 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:01.633280 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.633287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.634040 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.634041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.635437 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.635441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.635962 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.635963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.636123 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.636649 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:01.636661 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:01.638369 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:01.640238 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:01.640245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:01.640247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:01.640476 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:01.640758 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:01.640759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:01.640759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:01.640861 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:01.641220 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:01.641228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:01.641443 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 07:01:01.641571 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 07:01:01.641714 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260926 07:01:01.642665 18475 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36359
I20260926 07:01:01.642752 18475 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
I20260926 07:01:01.642938 18475 heartbeater.cc:511] Master 127.1.74.126:36359 requested a full tablet report, sending...
I20260926 07:01:01.643553 18284 ts_manager.cc:194] Registered new tserver with Master: 1e21546c4ef7475680811f549419645e (127.1.74.65:35567)
I20260926 07:01:01.644194 18284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:50871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 07:01:01.657929 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36359
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:01.738646 18480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:01.738854 18480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:01.738878 18480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:01.740226 18480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:01.740276 18480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:01.740334 18480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 07:01:01.741880 18480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:01.742133 18480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:01.742372 18480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:01.747526 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:01.747541 18486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:01.747527 18485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:01.747752 18480 server_base.cc:1034] running on GCE node
I20260926 07:01:01.747968 18480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:01.748216 18480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:01.749364 18480 hybrid_clock.cc:648] HybridClock initialized: now 1790406061749351 us; error 27 us; skew 500 ppm
Sep 26 07:01:01 dist-test-slave-sfvt krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:01.751884 18480 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 07:01:01.752388 18480 webserver.cc:460] Webserver started at http://127.1.74.66:37165/ using document root <none> and password file <none>
I20260926 07:01:01.752630 18480 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:01.752674 18480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:01.752826 18480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:01.753993 18480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "417ef1e13eef4f44a3d07a1d335799a6"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "20e003b26ba49bb81edaeea40f2f26da"
server_key_iv: "8e06d9a763925dec9f6f567bd840581c"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.754417 18480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "417ef1e13eef4f44a3d07a1d335799a6"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "20e003b26ba49bb81edaeea40f2f26da"
server_key_iv: "8e06d9a763925dec9f6f567bd840581c"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.755821 18480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:01.756518 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.756738 18480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:01.756850 18480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "417ef1e13eef4f44a3d07a1d335799a6"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "20e003b26ba49bb81edaeea40f2f26da"
server_key_iv: "8e06d9a763925dec9f6f567bd840581c"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.756925 18480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:01.766346 18480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:01.768221 18480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:01.768381 18480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:01.768637 18480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:01.768936 18480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:01.768991 18480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.769033 18480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:01.769060 18480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.774977 18480 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:34193
I20260926 07:01:01.775020 18608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:34193 every 8 connection(s)
I20260926 07:01:01.775415 18480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:01:01 dist-test-slave-sfvt krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:01.782698 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 18480
I20260926 07:01:01.782799 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:01.782965 1321 external_mini_cluster.cc:1428] Setting key 0aca2998418eb19234f0c48e25050cf0
I20260926 07:01:01.785003 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.776162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:52587 (local address 127.1.74.126:36359)
0926 07:01:01.776251 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:01.776254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:01.776392 (+ 138us) server_negotiation.cc:407] Connection header received
0926 07:01:01.776923 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:01.776924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:01.776935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:01.777000 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:01.777943 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.778457 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.779593 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.779780 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.779839 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.781887 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:01.781888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:01.781888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:01.781894 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:01.783572 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.784029 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.784030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.784030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.784095 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.784355 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.784356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.784356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.784476 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:01.784552 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:01.784825 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 07:01:01.784907 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:01:01.785215 18611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.776112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36359 (local address 127.1.74.66:52587)
0926 07:01:01.776274 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:01.776309 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:01.776808 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:01.777060 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:01.777063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:01.777645 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:01.777838 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.777843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.778566 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.778567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.779495 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.779498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.779855 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.779855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.779960 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.780431 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:01.780439 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:01.781803 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:01.783686 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:01.783695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:01.783697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:01.783935 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:01.784174 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:01.784175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:01.784175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:01.784281 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:01.784549 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:01.784555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:01.784695 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 07:01:01.784805 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:01:01.784970 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260926 07:01:01.785667 18609 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36359
I20260926 07:01:01.785782 18609 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:01.785960 18609 heartbeater.cc:511] Master 127.1.74.126:36359 requested a full tablet report, sending...
I20260926 07:01:01.786329 18284 ts_manager.cc:194] Registered new tserver with Master: 417ef1e13eef4f44a3d07a1d335799a6 (127.1.74.66:34193)
I20260926 07:01:01.786710 18284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:52587
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 07:01:01.808005 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36359
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:01.889168 18614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:01.889376 18614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:01.889400 18614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:01.890605 18614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:01.890654 18614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:01.890681 18614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 07:01:01.892189 18614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:36359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:01.892448 18614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:01.892697 18614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:01.897760 18622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:01.897745 18619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:01.897745 18620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:01.897957 18614 server_base.cc:1034] running on GCE node
I20260926 07:01:01.898166 18614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:01.898461 18614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:01.899628 18614 hybrid_clock.cc:648] HybridClock initialized: now 1790406061899615 us; error 28 us; skew 500 ppm
Sep 26 07:01:01 dist-test-slave-sfvt krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:01.902151 18614 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 07:01:01.902640 18614 webserver.cc:460] Webserver started at http://127.1.74.67:35737/ using document root <none> and password file <none>
I20260926 07:01:01.902881 18614 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:01.902926 18614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:01.903103 18614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:01.904218 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "1559236281e44507b8af565788a950c8"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "4358bb9da7b287088fe2448da0672b93"
server_key_iv: "20059c6b6a45378163c4bf104af5a95a"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.904613 18614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "1559236281e44507b8af565788a950c8"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "4358bb9da7b287088fe2448da0672b93"
server_key_iv: "20059c6b6a45378163c4bf104af5a95a"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.905921 18614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:01.906749 18629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.906946 18614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:01.907051 18614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "1559236281e44507b8af565788a950c8"
format_stamp: "Formatted at 2026-09-26 07:01:01 on dist-test-slave-sfvt"
server_key: "4358bb9da7b287088fe2448da0672b93"
server_key_iv: "20059c6b6a45378163c4bf104af5a95a"
server_key_version: "encryptionkey@0"
I20260926 07:01:01.907125 18614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:01.928892 18614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:01.931191 18614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:01.931407 18614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:01.931670 18614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:01.932020 18614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:01.932055 18614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.932078 18614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:01.932127 18614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:01.938164 18614 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:40373
I20260926 07:01:01.938210 18742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:40373 every 8 connection(s)
I20260926 07:01:01.938534 18614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 07:01:01.942922 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 18614
I20260926 07:01:01.943050 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:01.943238 1321 external_mini_cluster.cc:1428] Setting key 697291b78d98ad22a5c86ea78a4d01b9
Sep 26 07:01:01 dist-test-slave-sfvt krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406061, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:01.948207 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.939293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:39115 (local address 127.1.74.126:36359)
0926 07:01:01.939356 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:01.939358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:01.939599 (+ 241us) server_negotiation.cc:407] Connection header received
0926 07:01:01.940127 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:01.940129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:01.940141 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:01.940206 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:01.941180 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.941689 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.942762 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.942975 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.943053 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.945574 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:01.945575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:01.945576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:01.945584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:01.946778 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.947193 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.947195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.947195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.947277 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.947533 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.947534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.947535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.947664 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:01.947737 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:01.948025 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 07:01:01.948108 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:01:01.948416 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.939262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:36359 (local address 127.1.74.67:39115)
0926 07:01:01.939464 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:01.939505 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:01.940018 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:01.940283 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:01.940290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:01.940863 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:01.941074 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.941079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.941791 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.941792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.942665 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.942667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.943073 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.943074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.943205 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.943828 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:01.943838 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:01.945463 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:01.946866 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:01.946872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:01.946875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:01.947113 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:01.947343 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:01.947344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:01.947344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:01.947457 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:01.947738 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:01.947745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:01.947890 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 07:01:01.948011 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 07:01:01.948176 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260926 07:01:01.949002 18743 heartbeater.cc:347] Connected to a master server at 127.1.74.126:36359
I20260926 07:01:01.949086 18743 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:01.949273 18743 heartbeater.cc:511] Master 127.1.74.126:36359 requested a full tablet report, sending...
I20260926 07:01:01.949692 18284 ts_manager.cc:194] Registered new tserver with Master: 1559236281e44507b8af565788a950c8 (127.1.74.67:40373)
I20260926 07:01:01.950182 18284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:39115
I20260926 07:01:01.955770 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:01:01.963001 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.957219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.1.74.126:36359)
0926 07:01:01.957281 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:01.957283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:01.957371 (+ 88us) server_negotiation.cc:407] Connection header received
0926 07:01:01.957426 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:01.957427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:01.957435 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:01.957488 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:01.958022 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.958493 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.959215 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.959465 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.959526 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.960906 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:01.960906 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:01.960907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:01.960912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:01.962101 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.962387 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.962388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.962388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.962440 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:01.962563 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:01.962563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:01.962563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:01.962699 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:01.962770 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:01.962873 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 07:01:01.962930 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:01:01.964732 18284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:01.966305 18284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:01.972721 18751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.968593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:35567 (local address 127.0.0.1:57474)
0926 07:01:01.969049 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:01.969059 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:01.969106 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:01.969369 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:01.969370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:01.969372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:01.969415 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.969417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.970335 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.970336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.971660 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.971663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.972400 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.972401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.972481 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.972497 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:01:01.972571 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:01:01.972636 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":57,"threads_started":1}
I20260926 07:01:01.972733 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.968786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.1.74.65:35567)
0926 07:01:01.968975 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:01.969002 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:01.969158 (+ 156us) server_negotiation.cc:407] Connection header received
0926 07:01:01.969198 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:01.969201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:01.969216 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:01.969289 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:01.969491 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.970210 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.971760 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.972309 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.972401 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.972468 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:01.972603 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 07:01:01.972643 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260926 07:01:01.972920 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.968582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:40373 (local address 127.0.0.1:57500)
0926 07:01:01.968749 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:01.968791 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:01.968863 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:01.969174 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:01.969176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:01.969178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:01.969256 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.969258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.970662 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.970663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.971640 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.971643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.972669 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.972670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.972750 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.972753 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:01.972798 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 07:01:01.972846 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":28,"thread_start_us":57,"threads_started":1}
I20260926 07:01:01.973165 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.968814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.1.74.67:40373)
0926 07:01:01.969002 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:01.969007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:01.969019 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:01:01.969060 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:01.969062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:01.969079 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:01.969169 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:01.969418 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.970541 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.971739 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.972497 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.972573 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.972631 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:01.973035 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 07:01:01.973083 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260926 07:01:01.973574 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.968769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.1.74.66:34193)
0926 07:01:01.969005 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:01.969008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:01.969019 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:01:01.969091 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:01.969093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:01.969110 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:01.969179 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:01.969396 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.970571 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.971866 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.972531 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.972590 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.972636 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:01.973429 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 07:01:01.973468 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":105,"threads_started":1}
I20260926 07:01:01.973779 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:01.968582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:34193 (local address 127.0.0.1:50440)
0926 07:01:01.968975 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:01.968984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:01.969024 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:01.969260 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:01.969261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:01.969264 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:01.969317 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.969319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.970672 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.970673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.971784 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:01.971786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:01.972669 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:01.972670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:01.972746 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:01.972749 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:01.972798 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 07:01:01.973698 (+ 900us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":35,"thread_start_us":63,"threads_started":1}
I20260926 07:01:01.973666 18409 tablet_service.cc:1467] Processing CreateTablet for tablet 5847e9624ea140d8a4873e9bd3349829 (DEFAULT_TABLE table=test-table [id=db84580b7dcf4800a782a4620cd142ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:01.973773 18677 tablet_service.cc:1467] Processing CreateTablet for tablet 5847e9624ea140d8a4873e9bd3349829 (DEFAULT_TABLE table=test-table [id=db84580b7dcf4800a782a4620cd142ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:01.974043 18677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5847e9624ea140d8a4873e9bd3349829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:01.974043 18409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5847e9624ea140d8a4873e9bd3349829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:01.974399 18543 tablet_service.cc:1467] Processing CreateTablet for tablet 5847e9624ea140d8a4873e9bd3349829 (DEFAULT_TABLE table=test-table [id=db84580b7dcf4800a782a4620cd142ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:01.974674 18543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5847e9624ea140d8a4873e9bd3349829. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:01.977442 18757 tablet_bootstrap.cc:492] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: Bootstrap starting.
I20260926 07:01:01.978127 18757 tablet_bootstrap.cc:654] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:01.978639 18757 log.cc:824] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:01.978963 18758 tablet_bootstrap.cc:492] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e: Bootstrap starting.
I20260926 07:01:01.979424 18759 tablet_bootstrap.cc:492] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: Bootstrap starting.
I20260926 07:01:01.979523 18757 tablet_bootstrap.cc:492] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: No bootstrap required, opened a new log
I20260926 07:01:01.979591 18757 ts_tablet_manager.cc:1397] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:01.979684 18758 tablet_bootstrap.cc:654] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:01.980055 18759 tablet_bootstrap.cc:654] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:01.980255 18758 log.cc:824] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:01.980623 18759 log.cc:824] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:01.981019 18758 tablet_bootstrap.cc:492] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e: No bootstrap required, opened a new log
I20260926 07:01:01.981104 18758 ts_tablet_manager.cc:1397] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:01.981683 18757 raft_consensus.cc:348] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } }
I20260926 07:01:01.981848 18757 raft_consensus.cc:374] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:01.981891 18757 raft_consensus.cc:729] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417ef1e13eef4f44a3d07a1d335799a6, State: Initialized, Role: FOLLOWER
I20260926 07:01:01.982007 18757 consensus_queue.cc:260] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } }
I20260926 07:01:01.982285 18757 ts_tablet_manager.cc:1428] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:01.982348 18609 heartbeater.cc:503] Master 127.1.74.126:36359 was elected leader, sending a full tablet report...
I20260926 07:01:01.982788 18758 raft_consensus.cc:348] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } }
I20260926 07:01:01.982916 18758 raft_consensus.cc:374] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:01.982939 18758 raft_consensus.cc:729] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e21546c4ef7475680811f549419645e, State: Initialized, Role: FOLLOWER
I20260926 07:01:01.983083 18758 consensus_queue.cc:260] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } }
I20260926 07:01:01.983413 18758 ts_tablet_manager.cc:1428] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:01:01.983421 18475 heartbeater.cc:503] Master 127.1.74.126:36359 was elected leader, sending a full tablet report...
I20260926 07:01:01.983944 18759 tablet_bootstrap.cc:492] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: No bootstrap required, opened a new log
I20260926 07:01:01.984038 18759 ts_tablet_manager.cc:1397] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 07:01:01.985651 18759 raft_consensus.cc:348] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } }
I20260926 07:01:01.985775 18759 raft_consensus.cc:374] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:01.985796 18759 raft_consensus.cc:729] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1559236281e44507b8af565788a950c8, State: Initialized, Role: FOLLOWER
I20260926 07:01:01.985920 18759 consensus_queue.cc:260] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } }
I20260926 07:01:01.986176 18759 ts_tablet_manager.cc:1428] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:01.986224 18743 heartbeater.cc:503] Master 127.1.74.126:36359 was elected leader, sending a full tablet report...
W20260926 07:01:02.026013 18610 tablet.cc:2369] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:01:02.123759 18764 raft_consensus.cc:482] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:02.123893 18764 raft_consensus.cc:504] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } }
I20260926 07:01:02.124241 18764 leader_election.cc:288] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 417ef1e13eef4f44a3d07a1d335799a6 (127.1.74.66:34193), 1559236281e44507b8af565788a950c8 (127.1.74.67:40373)
I20260926 07:01:02.127144 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.124542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:51107 (local address 127.1.74.67:40373)
0926 07:01:02.124614 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.124617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.124626 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:02.124661 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.124662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.124671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.124756 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:02.124935 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.125415 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.126387 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.126768 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.126851 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.126861 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.127037 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.127077 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:01:02.127144 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.124456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:40373 (local address 127.1.74.65:51107)
0926 07:01:02.124529 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.124542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.124588 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.124762 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.124763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.124766 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:02.124848 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.124850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.125509 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.125510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.126287 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.126289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.126861 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.126862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.126953 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.126956 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.127016 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.127061 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":20}
I20260926 07:01:02.127233 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.124537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:54575 (local address 127.1.74.66:34193)
0926 07:01:02.124595 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.124597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.124767 (+ 170us) server_negotiation.cc:407] Connection header received
0926 07:01:02.124821 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.124822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.124836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.124909 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:02.125080 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.125537 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.126473 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.126848 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.126936 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.126947 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.127097 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.127135 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:01:02.127310 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.124456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:34193 (local address 127.1.74.65:54575)
0926 07:01:02.124680 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.124688 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.124765 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.124925 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.124926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.124928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:02.125005 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.125006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.125626 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.125627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.126375 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.126377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.126949 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.126950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.127023 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.127027 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.127091 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.127131 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":110,"thread_start_us":75,"threads_started":1}
I20260926 07:01:02.127629 18563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5847e9624ea140d8a4873e9bd3349829" candidate_uuid: "1e21546c4ef7475680811f549419645e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417ef1e13eef4f44a3d07a1d335799a6" is_pre_election: true
I20260926 07:01:02.127611 18697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5847e9624ea140d8a4873e9bd3349829" candidate_uuid: "1e21546c4ef7475680811f549419645e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1559236281e44507b8af565788a950c8" is_pre_election: true
I20260926 07:01:02.127784 18563 raft_consensus.cc:2454] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e21546c4ef7475680811f549419645e in term 0.
I20260926 07:01:02.127784 18697 raft_consensus.cc:2454] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e21546c4ef7475680811f549419645e in term 0.
I20260926 07:01:02.128072 18362 leader_election.cc:302] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e21546c4ef7475680811f549419645e, 417ef1e13eef4f44a3d07a1d335799a6; no voters:
I20260926 07:01:02.128228 18764 raft_consensus.cc:2781] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:02.128309 18764 raft_consensus.cc:482] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:02.128340 18764 raft_consensus.cc:3037] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:02.129112 18764 raft_consensus.cc:504] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } }
I20260926 07:01:02.129271 18764 leader_election.cc:288] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [CANDIDATE]: Term 1 election: Requested vote from peers 417ef1e13eef4f44a3d07a1d335799a6 (127.1.74.66:34193), 1559236281e44507b8af565788a950c8 (127.1.74.67:40373)
I20260926 07:01:02.129415 18563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5847e9624ea140d8a4873e9bd3349829" candidate_uuid: "1e21546c4ef7475680811f549419645e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417ef1e13eef4f44a3d07a1d335799a6"
I20260926 07:01:02.129437 18697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5847e9624ea140d8a4873e9bd3349829" candidate_uuid: "1e21546c4ef7475680811f549419645e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1559236281e44507b8af565788a950c8"
I20260926 07:01:02.129495 18563 raft_consensus.cc:3037] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:02.129552 18697 raft_consensus.cc:3037] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:02.130224 18563 raft_consensus.cc:2454] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e21546c4ef7475680811f549419645e in term 1.
I20260926 07:01:02.130398 18362 leader_election.cc:302] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e21546c4ef7475680811f549419645e, 417ef1e13eef4f44a3d07a1d335799a6; no voters:
I20260926 07:01:02.130427 18697 raft_consensus.cc:2454] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e21546c4ef7475680811f549419645e in term 1.
I20260926 07:01:02.130493 18764 raft_consensus.cc:2781] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:02.130654 18764 raft_consensus.cc:686] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 1 LEADER]: Becoming Leader. State: Replica: 1e21546c4ef7475680811f549419645e, State: Running, Role: LEADER
I20260926 07:01:02.130775 18764 consensus_queue.cc:237] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } }
W20260926 07:01:02.131371 18476 tablet.cc:2369] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:01:02.131639 18284 catalog_manager.cc:5495] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e21546c4ef7475680811f549419645e (127.1.74.65). New cstate: current_term: 1 leader_uuid: "1e21546c4ef7475680811f549419645e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e21546c4ef7475680811f549419645e" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 } health_report { overall_health: UNKNOWN } } }
I20260926 07:01:02.155521 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.152924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.1.74.65:35567)
0926 07:01:02.153020 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.153022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.153067 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:01:02.153113 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.153114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.153139 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.153208 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:02.153331 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.153868 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.154673 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.154864 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.154939 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.155106 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:02.155217 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:02.155337 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.155420 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.155440 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 07:01:02.157449 18563 raft_consensus.cc:1261] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [term 1 FOLLOWER]: Refusing update from remote peer 1e21546c4ef7475680811f549419645e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:02.157476 18697 raft_consensus.cc:1261] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [term 1 FOLLOWER]: Refusing update from remote peer 1e21546c4ef7475680811f549419645e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:02.157840 18764 consensus_queue.cc:1035] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [LEADER]: Connected to new peer: Peer: permanent_uuid: "417ef1e13eef4f44a3d07a1d335799a6" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:02.158020 18767 consensus_queue.cc:1035] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1559236281e44507b8af565788a950c8" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 40373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:02.160758 18771 mvcc.cc:205] Tried to move back new op lower bound from 7333503230594240512 to 7333503230488432640. Current Snapshot: MvccSnapshot[applied={T|T < 7333503230594240512}]
I20260926 07:01:02.161392 18772 mvcc.cc:205] Tried to move back new op lower bound from 7333503230594240512 to 7333503230488432640. Current Snapshot: MvccSnapshot[applied={T|T < 7333503230594240512}]
I20260926 07:01:02.163513 18283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:02.163615 18283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:02.164407 18283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12c4d28b60a9479c983195cfbe77d499: Sending DeleteTablet for 3 replicas of tablet 5847e9624ea140d8a4873e9bd3349829
I20260926 07:01:02.164763 18677 tablet_service.cc:1514] Processing DeleteTablet for tablet 5847e9624ea140d8a4873e9bd3349829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:02 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57500
I20260926 07:01:02.164750 18543 tablet_service.cc:1514] Processing DeleteTablet for tablet 5847e9624ea140d8a4873e9bd3349829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:02 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:50440
I20260926 07:01:02.164968 18409 tablet_service.cc:1514] Processing DeleteTablet for tablet 5847e9624ea140d8a4873e9bd3349829 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:02 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:57474
I20260926 07:01:02.165206 18782 tablet_replica.cc:330] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e: stopping tablet replica
I20260926 07:01:02.165304 18781 tablet_replica.cc:330] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: stopping tablet replica
I20260926 07:01:02.165321 18782 raft_consensus.cc:2229] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:01:02.165412 18781 raft_consensus.cc:2229] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:02.165442 18782 raft_consensus.cc:2258] T 5847e9624ea140d8a4873e9bd3349829 P 1e21546c4ef7475680811f549419645e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:02.165506 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 18346
I20260926 07:01:02.165526 18781 raft_consensus.cc:2258] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:02.166091 18780 tablet_replica.cc:330] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: stopping tablet replica
I20260926 07:01:02.166199 18780 raft_consensus.cc:2229] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:02.166304 18780 raft_consensus.cc:2258] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:02.166599 18781 ts_tablet_manager.cc:1905] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:02.166769 18780 ts_tablet_manager.cc:1905] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:02.168740 18780 ts_tablet_manager.cc:1918] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:02.168740 18781 ts_tablet_manager.cc:1918] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:02.168846 18780 log.cc:1196] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/5847e9624ea140d8a4873e9bd3349829
I20260926 07:01:02.168956 18781 log.cc:1196] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/5847e9624ea140d8a4873e9bd3349829
I20260926 07:01:02.169144 18780 ts_tablet_manager.cc:1939] T 5847e9624ea140d8a4873e9bd3349829 P 1559236281e44507b8af565788a950c8: Deleting consensus metadata
I20260926 07:01:02.169427 18781 ts_tablet_manager.cc:1939] T 5847e9624ea140d8a4873e9bd3349829 P 417ef1e13eef4f44a3d07a1d335799a6: Deleting consensus metadata
I20260926 07:01:02.169593 18271 catalog_manager.cc:4841] TS 1559236281e44507b8af565788a950c8 (127.1.74.67:40373): tablet 5847e9624ea140d8a4873e9bd3349829 (table test-table [id=db84580b7dcf4800a782a4620cd142ca]) successfully deleted
I20260926 07:01:02.169951 18268 catalog_manager.cc:4841] TS 417ef1e13eef4f44a3d07a1d335799a6 (127.1.74.66:34193): tablet 5847e9624ea140d8a4873e9bd3349829 (table test-table [id=db84580b7dcf4800a782a4620cd142ca]) successfully deleted
W20260926 07:01:02.172034 18270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:35567 (error 108)
I20260926 07:01:02.172276 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 18480
I20260926 07:01:02.172824 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.172382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:35567 (local address 127.0.0.1:57492)
0926 07:01:02.172749 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.172788 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:35567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344}
W20260926 07:01:02.173182 18270 catalog_manager.cc:4584] TS 1e21546c4ef7475680811f549419645e (127.1.74.65:35567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5847e9624ea140d8a4873e9bd3349829: Network error: Client connection negotiation failed: client connection to 127.1.74.65:35567: connect: Connection refused (error 111)
I20260926 07:01:02.177238 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 18614
I20260926 07:01:02.181708 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 18252
2026-09-26T07:01:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2032 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T07:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:02Z Disabled control of system clock
I20260926 07:01:02.209558 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46777
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:46777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:01:02.291477 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:02.291708 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:02.291730 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:02.292966 18787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:02.293015 18787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:02.293027 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:02.293040 18787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:02.293054 18787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:02.294515 18787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:46777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:02.294795 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:02.295105 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:02.300122 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:02.300251 18787 server_base.cc:1034] running on GCE node
W20260926 07:01:02.300122 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:02.300196 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:02.300652 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:02.300962 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:02.302132 18787 hybrid_clock.cc:648] HybridClock initialized: now 1790406062302120 us; error 47 us; skew 500 ppm
I20260926 07:01:02.303406 18787 webserver.cc:460] Webserver started at http://127.1.74.126:42709/ using document root <none> and password file <none>
I20260926 07:01:02.303663 18787 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:02.303730 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:02.303889 18787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:02.304976 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "94ade940353146f49b5156e7409e004c"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "24a545d562d5bb0d1ebbcdbae29256e7"
server_key_iv: "674d6927e99de0595303775b67afb0b0"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.305347 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "94ade940353146f49b5156e7409e004c"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "24a545d562d5bb0d1ebbcdbae29256e7"
server_key_iv: "674d6927e99de0595303775b67afb0b0"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.306655 18787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:02.307447 18801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.307621 18787 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:02.307725 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "94ade940353146f49b5156e7409e004c"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "24a545d562d5bb0d1ebbcdbae29256e7"
server_key_iv: "674d6927e99de0595303775b67afb0b0"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.307799 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:02.331750 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:02.332111 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:02.332274 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:02.336261 18787 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:46777
I20260926 07:01:02.336308 18853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:46777 every 8 connection(s)
I20260926 07:01:02.336664 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:02.337632 18854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:02.340099 18854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c: Bootstrap starting.
I20260926 07:01:02.340694 18854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:02.341009 18854 log.cc:824] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:02.341697 18854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c: No bootstrap required, opened a new log
I20260926 07:01:02.343457 18854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ade940353146f49b5156e7409e004c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46777 } }
I20260926 07:01:02.343600 18854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:02.343623 18854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ade940353146f49b5156e7409e004c, State: Initialized, Role: FOLLOWER
I20260926 07:01:02.343804 18854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ade940353146f49b5156e7409e004c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46777 } }
I20260926 07:01:02.343887 18854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:02.343933 18854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:02.343982 18854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:02.344338 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 18787
I20260926 07:01:02.344430 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:02.344558 18854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ade940353146f49b5156e7409e004c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46777 } }
I20260926 07:01:02.344627 1321 external_mini_cluster.cc:1428] Setting key 0e8f6fff48ff91273491e790c8b87ccd
I20260926 07:01:02.344676 18854 leader_election.cc:302] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94ade940353146f49b5156e7409e004c; no voters:
I20260926 07:01:02.344867 18854 leader_election.cc:288] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:02.344918 18857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:02.345110 18857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [term 1 LEADER]: Becoming Leader. State: Replica: 94ade940353146f49b5156e7409e004c, State: Running, Role: LEADER
I20260926 07:01:02.345240 18854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:02.345232 18857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ade940353146f49b5156e7409e004c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46777 } }
I20260926 07:01:02.345842 18859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94ade940353146f49b5156e7409e004c. Latest consensus state: current_term: 1 leader_uuid: "94ade940353146f49b5156e7409e004c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ade940353146f49b5156e7409e004c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46777 } } }
I20260926 07:01:02.346095 18859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:02.346010 18858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94ade940353146f49b5156e7409e004c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94ade940353146f49b5156e7409e004c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46777 } } }
I20260926 07:01:02.346556 18858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:02.346663 18868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:02.347155 18868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:02.348659 18868 catalog_manager.cc:1347] Generated new cluster ID: d822a6826f1440e2ba8dead03f0280dc
I20260926 07:01:02.348716 18868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:02.350965 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.345306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.1.74.126:46777)
0926 07:01:02.345723 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.345732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.345766 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:01:02.345907 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.345915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.346004 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.346437 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:02.346566 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.348064 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.349034 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.349462 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.349597 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.350320 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:02.350324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:02.350324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:02.350504 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:02.350537 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:02.350547 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:02.350627 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.350708 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.350796 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":71,"threads_started":1}
I20260926 07:01:02.373171 18868 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:02.373961 18868 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:02.381448 18868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c: Generated new TSK 0
I20260926 07:01:02.381680 18868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:02.389389 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46777
--builtin_ntp_servers=127.1.74.84:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:01:02.483682 18878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:02.483925 18878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:02.483963 18878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:02.485181 18878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:02.485278 18878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:02.485323 18878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 07:01:02.486682 18878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:02.486956 18878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:02.487294 18878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:02.493279 18886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:02.493296 18884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:02.493299 18883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:02.493476 18878 server_base.cc:1034] running on GCE node
I20260926 07:01:02.493794 18878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:02.494043 18878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:02.495203 18878 hybrid_clock.cc:648] HybridClock initialized: now 1790406062495191 us; error 30 us; skew 500 ppm
I20260926 07:01:02.496570 18878 webserver.cc:460] Webserver started at http://127.1.74.65:42641/ using document root <none> and password file <none>
I20260926 07:01:02.496842 18878 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:02.496910 18878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:02.497066 18878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:02.498124 18878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "09ad70ff9d374fbe947f7ac6b8fca907"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "8214b804a4da0547d817be5afb727201"
server_key_iv: "f01874fcbad7f939b281706b21bd4660"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.498482 18878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "09ad70ff9d374fbe947f7ac6b8fca907"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "8214b804a4da0547d817be5afb727201"
server_key_iv: "f01874fcbad7f939b281706b21bd4660"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.499939 18878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 07:01:02.500736 18892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.501008 18878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:02.501116 18878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "09ad70ff9d374fbe947f7ac6b8fca907"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "8214b804a4da0547d817be5afb727201"
server_key_iv: "f01874fcbad7f939b281706b21bd4660"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.501204 18878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:02.511716 18878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:02.512028 18878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:02.512178 18878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:02.512408 18878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:02.512730 18878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:02.512786 18878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.512827 18878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:02.512857 18878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.518563 18878 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:37059
I20260926 07:01:02.518642 19005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:37059 every 8 connection(s)
I20260926 07:01:02.519006 18878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 07:01:02.524031 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.519804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:55845 (local address 127.1.74.126:46777)
0926 07:01:02.519887 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.519889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.520141 (+ 252us) server_negotiation.cc:407] Connection header received
0926 07:01:02.520627 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.520627 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.520638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.520695 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:02.521352 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.521833 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.522774 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.522950 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.522999 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.523809 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:02.523810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:02.523811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:02.523815 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:02.523827 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:02.523831 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:02.523879 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.523946 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.523971 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:01:02.524224 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.519802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:46777 (local address 127.1.74.65:55845)
0926 07:01:02.520008 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.520048 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.520543 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.520759 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.520763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.520772 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:02.521282 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.521286 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.521916 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.521916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.522699 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.522701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.523024 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.523025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.523132 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.523680 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:01:02.523681 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:02.523710 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:01:02.523733 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:02.523881 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:02.523888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:02.523890 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.523936 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.524012 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260926 07:01:02.524842 19006 heartbeater.cc:347] Connected to a master server at 127.1.74.126:46777
I20260926 07:01:02.524945 19006 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:02.525023 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 18878
I20260926 07:01:02.525138 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:02.525156 19006 heartbeater.cc:511] Master 127.1.74.126:46777 requested a full tablet report, sending...
I20260926 07:01:02.525367 1321 external_mini_cluster.cc:1428] Setting key a83e922e8ef02f6df23d9470d158582b
I20260926 07:01:02.525772 18818 ts_manager.cc:194] Registered new tserver with Master: 09ad70ff9d374fbe947f7ac6b8fca907 (127.1.74.65:37059)
I20260926 07:01:02.526451 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46777
--builtin_ntp_servers=127.1.74.84:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:01:02.526654 18818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.65:55845
W20260926 07:01:02.607034 19009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:02.607319 19009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:02.607372 19009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:02.608556 19009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:02.608623 19009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:02.608685 19009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 07:01:02.610105 19009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:02.610376 19009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:02.610733 19009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:02.615789 19015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:02.615814 19014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:02.615814 19017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:02.616302 19009 server_base.cc:1034] running on GCE node
I20260926 07:01:02.616496 19009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:02.616750 19009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:02.617892 19009 hybrid_clock.cc:648] HybridClock initialized: now 1790406062617876 us; error 33 us; skew 500 ppm
I20260926 07:01:02.619242 19009 webserver.cc:460] Webserver started at http://127.1.74.66:39261/ using document root <none> and password file <none>
I20260926 07:01:02.619551 19009 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:02.619599 19009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:02.619768 19009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:02.620857 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "e36379c2540c43b79db6f4ca47ab7bda"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "cd1685281969ba67fe5bad0ecaa47819"
server_key_iv: "687145fbaf69dc6d1e5e3a8ec1f74008"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.621217 19009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "e36379c2540c43b79db6f4ca47ab7bda"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "cd1685281969ba67fe5bad0ecaa47819"
server_key_iv: "687145fbaf69dc6d1e5e3a8ec1f74008"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.622457 19009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:02.623116 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.623322 19009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:02.623416 19009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "e36379c2540c43b79db6f4ca47ab7bda"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "cd1685281969ba67fe5bad0ecaa47819"
server_key_iv: "687145fbaf69dc6d1e5e3a8ec1f74008"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.623498 19009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:02.665715 19009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:02.666074 19009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:02.666245 19009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:02.666492 19009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:02.666836 19009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:02.666893 19009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.666934 19009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:02.666975 19009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.672782 19009 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:43057
I20260926 07:01:02.672825 19136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:43057 every 8 connection(s)
I20260926 07:01:02.673198 19009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 07:01:02.678701 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.673981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:49625 (local address 127.1.74.126:46777)
0926 07:01:02.674053 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.674055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.674301 (+ 246us) server_negotiation.cc:407] Connection header received
0926 07:01:02.674861 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.674862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.674874 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.674927 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:02.675691 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.676192 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.677268 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.677474 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.677553 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.678387 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:02.678388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:02.678389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:02.678394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:02.678410 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:02.678416 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:02.678475 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.678578 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.678607 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:01:02.678948 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.673965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:46777 (local address 127.1.74.66:49625)
0926 07:01:02.674175 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.674210 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.674746 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.675003 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.675007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.675015 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:02.675586 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.675590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.676296 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.676297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.677125 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.677127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.677560 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.677561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.677704 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.678234 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:01:02.678235 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:02.678264 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:01:02.678287 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:02.678490 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:02.678495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:02.678498 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.678581 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.678711 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260926 07:01:02.679512 19137 heartbeater.cc:347] Connected to a master server at 127.1.74.126:46777
I20260926 07:01:02.679612 19137 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:02.679773 19137 heartbeater.cc:511] Master 127.1.74.126:46777 requested a full tablet report, sending...
I20260926 07:01:02.680140 18818 ts_manager.cc:194] Registered new tserver with Master: e36379c2540c43b79db6f4ca47ab7bda (127.1.74.66:43057)
I20260926 07:01:02.680501 18818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.66:49625
I20260926 07:01:02.681649 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 19009
I20260926 07:01:02.681737 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:02.681882 1321 external_mini_cluster.cc:1428] Setting key e73caf023343904dd4718724e08e5233
I20260926 07:01:02.682797 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46777
--builtin_ntp_servers=127.1.74.84:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:01:02.763459 19140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:02.763705 19140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:02.763741 19140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:02.764916 19140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:02.764991 19140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:02.765033 19140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 07:01:02.766374 19140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:46777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:02.766654 19140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:02.766953 19140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:02.771968 19145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:02.772126 19140 server_base.cc:1034] running on GCE node
W20260926 07:01:02.771986 19148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:02.771984 19146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:02.772377 19140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:02.772578 19140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:02.773751 19140 hybrid_clock.cc:648] HybridClock initialized: now 1790406062773727 us; error 35 us; skew 500 ppm
I20260926 07:01:02.775059 19140 webserver.cc:460] Webserver started at http://127.1.74.67:40487/ using document root <none> and password file <none>
I20260926 07:01:02.775365 19140 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:02.775413 19140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:02.775573 19140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:02.776669 19140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "c5b9b00ecb544de5a30b9080b390f277"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "f219566d06308b2ac95fc1e7e36f303c"
server_key_iv: "eb396d0afd116fe6d192599c2d4a8eab"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.777032 19140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "c5b9b00ecb544de5a30b9080b390f277"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "f219566d06308b2ac95fc1e7e36f303c"
server_key_iv: "eb396d0afd116fe6d192599c2d4a8eab"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.778368 19140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:02.779174 19154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.779381 19140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:02.779472 19140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "c5b9b00ecb544de5a30b9080b390f277"
format_stamp: "Formatted at 2026-09-26 07:01:02 on dist-test-slave-sfvt"
server_key: "f219566d06308b2ac95fc1e7e36f303c"
server_key_iv: "eb396d0afd116fe6d192599c2d4a8eab"
server_key_version: "encryptionkey@0"
I20260926 07:01:02.779556 19140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:02.798063 19140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:02.798389 19140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:02.798537 19140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:02.798782 19140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:02.799122 19140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:02.799157 19140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.799180 19140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:02.799229 19140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:02.805176 19140 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:41739
I20260926 07:01:02.805236 19267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:41739 every 8 connection(s)
I20260926 07:01:02.805552 19140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 07:01:02.807521 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 19140
I20260926 07:01:02.807636 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:02.807845 1321 external_mini_cluster.cc:1428] Setting key d8337c472c1aa100e375ebcdc9451a16
I20260926 07:01:02.811447 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.806359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:49041 (local address 127.1.74.126:46777)
0926 07:01:02.806429 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.806431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.806694 (+ 263us) server_negotiation.cc:407] Connection header received
0926 07:01:02.807241 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.807243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.807286 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.807345 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:02.808321 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.809005 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.809994 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.810225 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.810312 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.811131 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:02.811132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:02.811132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:02.811138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:02.811152 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:02.811157 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:02.811203 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.811335 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.811367 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:01:02.811674 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.806333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:46777 (local address 127.1.74.67:49041)
0926 07:01:02.806562 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.806607 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.807131 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.807484 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.807489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.807500 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:02.808225 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.808231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.809096 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.809096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.809883 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.809885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.810322 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.810323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.810464 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.810964 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:01:02.810965 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:02.810993 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:01:02.811016 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:02.811203 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:02.811208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:02.811211 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.811335 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.811441 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
I20260926 07:01:02.812173 19268 heartbeater.cc:347] Connected to a master server at 127.1.74.126:46777
I20260926 07:01:02.812325 19268 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:02.812533 19268 heartbeater.cc:511] Master 127.1.74.126:46777 requested a full tablet report, sending...
I20260926 07:01:02.812916 18818 ts_manager.cc:194] Registered new tserver with Master: c5b9b00ecb544de5a30b9080b390f277 (127.1.74.67:41739)
I20260926 07:01:02.813287 18818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.67:49041
I20260926 07:01:02.819955 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:01:02.823901 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.821330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.1.74.126:46777)
0926 07:01:02.821386 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.821389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.821452 (+ 63us) server_negotiation.cc:407] Connection header received
0926 07:01:02.821499 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.821500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.821510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.821568 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:02.821691 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.822158 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.822851 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.823024 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.823077 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.823717 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:02.823718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:02.823719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:02.823723 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:02.823737 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:02.823742 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:02.823783 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.823817 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.823840 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:01:02.825434 18818 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:02.827129 18818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:02.833344 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.829208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:37059 (local address 127.0.0.1:59250)
0926 07:01:02.829369 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.829418 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.829488 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.829701 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.829703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.829706 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:02.829770 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.829772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.830903 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.830905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.832088 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.832091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.833013 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.833014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.833098 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.833113 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.833191 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.833244 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260926 07:01:02.833667 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.829592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:41739 (local address 127.0.0.1:50976)
0926 07:01:02.830051 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.830062 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.830106 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.830414 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.830415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.830417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:02.830451 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.830452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.831673 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.831674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.832565 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.832568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.833432 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.833433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.833499 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.833503 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.833567 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.833605 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":65,"threads_started":1}
I20260926 07:01:02.833695 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.829856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.1.74.67:41739)
0926 07:01:02.830256 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.830261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.830270 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:02.830307 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.830309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.830325 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.830405 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:02.830509 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.831584 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.832655 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.833261 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.833321 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.833398 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.833583 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.833619 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":80,"threads_started":1}
I20260926 07:01:02.833806 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.829452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:43057 (local address 127.0.0.1:42416)
0926 07:01:02.829568 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.829578 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.829627 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.830149 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.830150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.830153 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:02.830198 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.830200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.831511 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.831513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.832571 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.832574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.833563 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.833564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.833623 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.833627 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.833674 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.833714 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
I20260926 07:01:02.834043 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.829769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.1.74.66:43057)
0926 07:01:02.829931 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.829935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.829944 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:02.829982 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.829983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.829998 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.830083 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:02.830261 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.831421 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.832665 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.833363 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.833427 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.833519 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.833914 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.833963 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260926 07:01:02.834043 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.829309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.1.74.65:37059)
0926 07:01:02.829487 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.829492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.829503 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:01:02.829587 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.829589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.829603 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.829696 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:02.829833 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.830826 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.832189 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.832928 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.833002 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.833102 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.833308 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.833352 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260926 07:01:02.834484 19202 tablet_service.cc:1467] Processing CreateTablet for tablet 2d93b025d2ac4230bef169c38755939e (DEFAULT_TABLE table=test-table [id=262ce50d1cfb43c2b905b59ef05ec252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:02.834718 19071 tablet_service.cc:1467] Processing CreateTablet for tablet 2d93b025d2ac4230bef169c38755939e (DEFAULT_TABLE table=test-table [id=262ce50d1cfb43c2b905b59ef05ec252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:02.834719 18940 tablet_service.cc:1467] Processing CreateTablet for tablet 2d93b025d2ac4230bef169c38755939e (DEFAULT_TABLE table=test-table [id=262ce50d1cfb43c2b905b59ef05ec252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:02.834854 19202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d93b025d2ac4230bef169c38755939e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:02.835038 19071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d93b025d2ac4230bef169c38755939e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:02.835038 18940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d93b025d2ac4230bef169c38755939e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:02.838426 19282 tablet_bootstrap.cc:492] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: Bootstrap starting.
I20260926 07:01:02.838443 19284 tablet_bootstrap.cc:492] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907: Bootstrap starting.
I20260926 07:01:02.839025 19284 tablet_bootstrap.cc:654] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:02.839025 19282 tablet_bootstrap.cc:654] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:02.839392 19282 log.cc:824] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:02.839419 19283 tablet_bootstrap.cc:492] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: Bootstrap starting.
I20260926 07:01:02.839427 19284 log.cc:824] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:02.840056 19283 tablet_bootstrap.cc:654] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:02.840145 19282 tablet_bootstrap.cc:492] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: No bootstrap required, opened a new log
I20260926 07:01:02.840154 19284 tablet_bootstrap.cc:492] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907: No bootstrap required, opened a new log
I20260926 07:01:02.840221 19284 ts_tablet_manager.cc:1397] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:02.840221 19282 ts_tablet_manager.cc:1397] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:02.840378 19283 log.cc:824] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:02.841261 19283 tablet_bootstrap.cc:492] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: No bootstrap required, opened a new log
I20260926 07:01:02.841352 19283 ts_tablet_manager.cc:1397] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:02.842837 19284 raft_consensus.cc:348] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } }
I20260926 07:01:02.842837 19282 raft_consensus.cc:348] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } }
I20260926 07:01:02.843020 19282 raft_consensus.cc:374] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:02.843020 19284 raft_consensus.cc:374] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:02.843057 19282 raft_consensus.cc:729] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b9b00ecb544de5a30b9080b390f277, State: Initialized, Role: FOLLOWER
I20260926 07:01:02.843067 19284 raft_consensus.cc:729] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ad70ff9d374fbe947f7ac6b8fca907, State: Initialized, Role: FOLLOWER
I20260926 07:01:02.843186 19282 consensus_queue.cc:260] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } }
I20260926 07:01:02.843225 19284 consensus_queue.cc:260] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } }
I20260926 07:01:02.843243 19283 raft_consensus.cc:348] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } }
I20260926 07:01:02.843451 19283 raft_consensus.cc:374] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:02.843479 19282 ts_tablet_manager.cc:1428] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 07:01:02.843479 19284 ts_tablet_manager.cc:1428] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 07:01:02.843495 19283 raft_consensus.cc:729] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36379c2540c43b79db6f4ca47ab7bda, State: Initialized, Role: FOLLOWER
I20260926 07:01:02.843596 19006 heartbeater.cc:503] Master 127.1.74.126:46777 was elected leader, sending a full tablet report...
I20260926 07:01:02.843640 19268 heartbeater.cc:503] Master 127.1.74.126:46777 was elected leader, sending a full tablet report...
I20260926 07:01:02.843642 19283 consensus_queue.cc:260] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } }
I20260926 07:01:02.843948 19283 ts_tablet_manager.cc:1428] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:02.844089 19137 heartbeater.cc:503] Master 127.1.74.126:46777 was elected leader, sending a full tablet report...
I20260926 07:01:02.853489 19289 raft_consensus.cc:482] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:02.853597 19289 raft_consensus.cc:504] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } }
I20260926 07:01:02.854001 19289 leader_election.cc:288] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e36379c2540c43b79db6f4ca47ab7bda (127.1.74.66:43057), c5b9b00ecb544de5a30b9080b390f277 (127.1.74.67:41739)
I20260926 07:01:02.857357 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.854173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:43057 (local address 127.1.74.65:32805)
0926 07:01:02.854236 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.854251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.854311 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.854450 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.854451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.854453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:02.854524 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.854526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.855611 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.855612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.856402 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.856404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.857101 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.857102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.857173 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.857177 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.857225 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.857259 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":30}
I20260926 07:01:02.857378 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.854247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:32805 (local address 127.1.74.66:43057)
0926 07:01:02.854319 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.854321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.854333 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:01:02.854363 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.854364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.854376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.854432 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:02.854693 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.855519 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.856493 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.857009 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.857072 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.857081 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.857256 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.857307 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:01:02.857434 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.854457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:44281 (local address 127.1.74.67:41739)
0926 07:01:02.854509 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.854511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.854518 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:01:02.854584 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.854585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.854595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.854652 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:02.854787 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.855488 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.856554 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.857069 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.857143 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.857152 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.857322 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.857360 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:01:02.857645 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.854302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:41739 (local address 127.1.74.65:44281)
0926 07:01:02.854465 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.854475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:02.854526 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:02.854657 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:02.854657 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:02.854659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:02.854717 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.854718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.855566 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.855566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.856421 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:02.856424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.857164 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.857165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:02.857243 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.857248 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:02.857305 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 07:01:02.857338 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":11,"thread_start_us":91,"threads_started":1}
I20260926 07:01:02.857764 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93b025d2ac4230bef169c38755939e" candidate_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36379c2540c43b79db6f4ca47ab7bda" is_pre_election: true
I20260926 07:01:02.857932 19091 raft_consensus.cc:2454] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ad70ff9d374fbe947f7ac6b8fca907 in term 0.
I20260926 07:01:02.858062 19222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93b025d2ac4230bef169c38755939e" candidate_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b9b00ecb544de5a30b9080b390f277" is_pre_election: true
I20260926 07:01:02.858174 19222 raft_consensus.cc:2454] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ad70ff9d374fbe947f7ac6b8fca907 in term 0.
I20260926 07:01:02.858376 18893 leader_election.cc:302] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ad70ff9d374fbe947f7ac6b8fca907, e36379c2540c43b79db6f4ca47ab7bda; no voters:
I20260926 07:01:02.858522 19289 raft_consensus.cc:2781] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:02.858631 19289 raft_consensus.cc:482] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:02.858668 19289 raft_consensus.cc:3037] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:02.859444 19289 raft_consensus.cc:504] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } }
I20260926 07:01:02.859592 19289 leader_election.cc:288] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [CANDIDATE]: Term 1 election: Requested vote from peers e36379c2540c43b79db6f4ca47ab7bda (127.1.74.66:43057), c5b9b00ecb544de5a30b9080b390f277 (127.1.74.67:41739)
I20260926 07:01:02.859776 19091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93b025d2ac4230bef169c38755939e" candidate_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36379c2540c43b79db6f4ca47ab7bda"
I20260926 07:01:02.859804 19222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93b025d2ac4230bef169c38755939e" candidate_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b9b00ecb544de5a30b9080b390f277"
I20260926 07:01:02.859871 19222 raft_consensus.cc:3037] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:02.859874 19091 raft_consensus.cc:3037] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:02.860589 19222 raft_consensus.cc:2454] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ad70ff9d374fbe947f7ac6b8fca907 in term 1.
I20260926 07:01:02.860670 19091 raft_consensus.cc:2454] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ad70ff9d374fbe947f7ac6b8fca907 in term 1.
I20260926 07:01:02.860749 18893 leader_election.cc:302] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ad70ff9d374fbe947f7ac6b8fca907, c5b9b00ecb544de5a30b9080b390f277; no voters:
I20260926 07:01:02.860853 19289 raft_consensus.cc:2781] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:02.861032 19289 raft_consensus.cc:686] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [term 1 LEADER]: Becoming Leader. State: Replica: 09ad70ff9d374fbe947f7ac6b8fca907, State: Running, Role: LEADER
I20260926 07:01:02.861137 19289 consensus_queue.cc:237] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } }
I20260926 07:01:02.862048 18818 catalog_manager.cc:5495] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ad70ff9d374fbe947f7ac6b8fca907 (127.1.74.65). New cstate: current_term: 1 leader_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ad70ff9d374fbe947f7ac6b8fca907" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 37059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 } health_report { overall_health: UNKNOWN } } }
I20260926 07:01:02.868304 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.866134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.1.74.65:37059)
0926 07:01:02.866189 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:02.866191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:02.866198 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:01:02.866276 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:02.866277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:02.866294 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:02.866351 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:02.866456 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.866898 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.867654 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:02.867828 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:02.867879 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:02.867995 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:02.868090 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:02.868149 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:02.868179 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 07:01:02.868196 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:01:02.869951 19222 raft_consensus.cc:1261] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [term 1 FOLLOWER]: Refusing update from remote peer 09ad70ff9d374fbe947f7ac6b8fca907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:02.869967 19091 raft_consensus.cc:1261] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [term 1 FOLLOWER]: Refusing update from remote peer 09ad70ff9d374fbe947f7ac6b8fca907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:02.870312 19289 consensus_queue.cc:1035] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b9b00ecb544de5a30b9080b390f277" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:02.870446 19292 consensus_queue.cc:1035] T 2d93b025d2ac4230bef169c38755939e P 09ad70ff9d374fbe947f7ac6b8fca907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36379c2540c43b79db6f4ca47ab7bda" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:02.873302 19297 mvcc.cc:205] Tried to move back new op lower bound from 7333503233513619456 to 7333503233480724480. Current Snapshot: MvccSnapshot[applied={T|T < 7333503233513619456}]
I20260926 07:01:02.873688 19296 mvcc.cc:205] Tried to move back new op lower bound from 7333503233513619456 to 7333503233480724480. Current Snapshot: MvccSnapshot[applied={T|T < 7333503233513619456 or (T in {7333503233513619456})}]
I20260926 07:01:02.875020 18817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:02.875113 18817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:02.875876 18817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94ade940353146f49b5156e7409e004c: Sending DeleteTablet for 3 replicas of tablet 2d93b025d2ac4230bef169c38755939e
I20260926 07:01:02.876252 18940 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93b025d2ac4230bef169c38755939e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:02 UTC) from {username='slave'} at 127.0.0.1:59250
I20260926 07:01:02.876252 19202 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93b025d2ac4230bef169c38755939e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:02 UTC) from {username='slave'} at 127.0.0.1:50976
I20260926 07:01:02.876470 19071 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93b025d2ac4230bef169c38755939e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:02 UTC) from {username='slave'} at 127.0.0.1:42416
I20260926 07:01:02.876513 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 18878
I20260926 07:01:02.878470 19307 tablet_replica.cc:330] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: stopping tablet replica
I20260926 07:01:02.878563 19307 raft_consensus.cc:2229] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:02.878652 19307 raft_consensus.cc:2258] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:02.879086 19307 ts_tablet_manager.cc:1905] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:02.879359 19305 tablet_replica.cc:330] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: stopping tablet replica
I20260926 07:01:02.879479 19305 raft_consensus.cc:2229] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:02.879602 19305 raft_consensus.cc:2258] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:02.879990 19305 ts_tablet_manager.cc:1905] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:02.881125 19307 ts_tablet_manager.cc:1918] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:02.881201 19307 log.cc:1196] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/2d93b025d2ac4230bef169c38755939e
I20260926 07:01:02.881268 19305 ts_tablet_manager.cc:1918] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:02.881335 19305 log.cc:1196] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/2d93b025d2ac4230bef169c38755939e
I20260926 07:01:02.881462 19307 ts_tablet_manager.cc:1939] T 2d93b025d2ac4230bef169c38755939e P e36379c2540c43b79db6f4ca47ab7bda: Deleting consensus metadata
I20260926 07:01:02.881626 19305 ts_tablet_manager.cc:1939] T 2d93b025d2ac4230bef169c38755939e P c5b9b00ecb544de5a30b9080b390f277: Deleting consensus metadata
I20260926 07:01:02.882041 18802 catalog_manager.cc:4841] TS c5b9b00ecb544de5a30b9080b390f277 (127.1.74.67:41739): tablet 2d93b025d2ac4230bef169c38755939e (table test-table [id=262ce50d1cfb43c2b905b59ef05ec252]) successfully deleted
I20260926 07:01:02.882143 18802 catalog_manager.cc:4841] TS e36379c2540c43b79db6f4ca47ab7bda (127.1.74.66:43057): tablet 2d93b025d2ac4230bef169c38755939e (table test-table [id=262ce50d1cfb43c2b905b59ef05ec252]) successfully deleted
W20260926 07:01:02.882939 18802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:37059 (error 108)
I20260926 07:01:02.883210 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 19009
I20260926 07:01:02.883435 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:02.883312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:37059 (local address 127.0.0.1:59270)
0926 07:01:02.883367 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:02.883400 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:37059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 07:01:02.883559 18802 catalog_manager.cc:4584] TS 09ad70ff9d374fbe947f7ac6b8fca907 (127.1.74.65:37059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d93b025d2ac4230bef169c38755939e: Network error: Client connection negotiation failed: client connection to 127.1.74.65:37059: connect: Connection refused (error 111)
I20260926 07:01:02.888466 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 19140
I20260926 07:01:02.893282 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 18787
2026-09-26T07:01:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (710 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:02 dist-test-slave-sfvt krb5kdc[19311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:02 dist-test-slave-sfvt krb5kdc[19311](info): set up 2 sockets
Sep 26 07:01:02 dist-test-slave-sfvt krb5kdc[19311](info): commencing operation
krb5kdc: starting...
W20260926 07:01:03.931035 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:04 dist-test-slave-sfvt krb5kdc[19311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:04.050251 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42399
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:42399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:04.130029 19324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:04.130240 19324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:04.130262 19324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:04.131407 19324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:04.131448 19324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:04.131462 19324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:04.131474 19324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:04.131487 19324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:04.132882 19324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:42399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:04.133074 19324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:04.133244 19324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:04.138018 19332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:04.138103 19330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:04.138141 19324 server_base.cc:1034] running on GCE node
W20260926 07:01:04.138026 19329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:04.138501 19324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:04.138751 19324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:04.139894 19324 hybrid_clock.cc:648] HybridClock initialized: now 1790406064139872 us; error 37 us; skew 500 ppm
Sep 26 07:01:04 dist-test-slave-sfvt krb5kdc[19311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:04.142273 19324 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:04.142735 19324 webserver.cc:460] Webserver started at http://127.1.74.126:35219/ using document root <none> and password file <none>
I20260926 07:01:04.142964 19324 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:04.143031 19324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:04.143181 19324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:04.144260 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "ec8f9d0634674863a214cd76257f35f1"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "38ddbf7eeffc556a1faafe5c3e431a03"
server_key_iv: "44c89fa94444c22b9dab4de63f3e04d2"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.144626 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "ec8f9d0634674863a214cd76257f35f1"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "38ddbf7eeffc556a1faafe5c3e431a03"
server_key_iv: "44c89fa94444c22b9dab4de63f3e04d2"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.145972 19324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:04.146672 19339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.146855 19324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:04.146972 19324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "ec8f9d0634674863a214cd76257f35f1"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "38ddbf7eeffc556a1faafe5c3e431a03"
server_key_iv: "44c89fa94444c22b9dab4de63f3e04d2"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.147054 19324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:04.156674 19324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:04.158435 19324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:04.158594 19324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:04.162592 19324 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:42399
I20260926 07:01:04.162624 19391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:42399 every 8 connection(s)
I20260926 07:01:04.162971 19324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:04.163830 19392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:04.164980 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 19324
I20260926 07:01:04.165093 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:04.165282 1321 external_mini_cluster.cc:1428] Setting key 12f79554c5d67f403580d47614693029
I20260926 07:01:04.166700 19392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1: Bootstrap starting.
I20260926 07:01:04.167462 19392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:04.167721 19392 log.cc:824] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:04.168356 19392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1: No bootstrap required, opened a new log
I20260926 07:01:04.170832 19392 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8f9d0634674863a214cd76257f35f1" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42399 } }
I20260926 07:01:04.171056 19392 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:04.171109 19392 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8f9d0634674863a214cd76257f35f1, State: Initialized, Role: FOLLOWER
I20260926 07:01:04.171284 19392 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8f9d0634674863a214cd76257f35f1" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42399 } }
I20260926 07:01:04.171384 19392 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:04.171420 19392 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:04.171466 19392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:04.172106 19392 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8f9d0634674863a214cd76257f35f1" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42399 } }
I20260926 07:01:04.172196 19392 leader_election.cc:302] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec8f9d0634674863a214cd76257f35f1; no voters:
Sep 26 07:01:04 dist-test-slave-sfvt krb5kdc[19311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:04.172382 19392 leader_election.cc:288] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:04.172422 19397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:04.172566 19397 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [term 1 LEADER]: Becoming Leader. State: Replica: ec8f9d0634674863a214cd76257f35f1, State: Running, Role: LEADER
I20260926 07:01:04.172647 19392 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:04.172667 19397 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8f9d0634674863a214cd76257f35f1" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42399 } }
I20260926 07:01:04.173012 19397 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec8f9d0634674863a214cd76257f35f1. Latest consensus state: current_term: 1 leader_uuid: "ec8f9d0634674863a214cd76257f35f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8f9d0634674863a214cd76257f35f1" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42399 } } }
I20260926 07:01:04.173092 19397 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:04.173231 19398 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec8f9d0634674863a214cd76257f35f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8f9d0634674863a214cd76257f35f1" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42399 } } }
I20260926 07:01:04.173305 19403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:04.173892 19403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:04.173905 19398 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:04.175814 19403 catalog_manager.cc:1347] Generated new cluster ID: ea675b9a51cb403aaf3d6a3cb2edc2b2
I20260926 07:01:04.175854 19403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:04.177392 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.165817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.1.74.126:42399)
0926 07:01:04.165986 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.165994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.166019 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:01:04.166447 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.166453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.166500 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.166617 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:04.167456 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.168512 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.169573 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.169969 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.170047 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.172847 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:04.172850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:04.172851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:04.172863 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:04.175691 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.176123 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.176126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.176127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.176239 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.176467 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.176468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.176468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.176767 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:04.176913 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.177032 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.177174 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260926 07:01:04.190686 19403 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:04.191202 19403 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:04.194983 19403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1: Generated new TSK 0
I20260926 07:01:04.195159 19403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 07:01:04.225771 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42399
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:04.306576 19418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:04.306788 19418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:04.306813 19418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:04.307914 19418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:04.307953 19418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:04.307981 19418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 07:01:04.309352 19418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:04.309542 19418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:04.309715 19418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:04.314697 19424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:04.314697 19423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:04.314746 19426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:04.314920 19418 server_base.cc:1034] running on GCE node
I20260926 07:01:04.315124 19418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:04.315407 19418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:04.316566 19418 hybrid_clock.cc:648] HybridClock initialized: now 1790406064316543 us; error 45 us; skew 500 ppm
Sep 26 07:01:04 dist-test-slave-sfvt krb5kdc[19311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:04.319242 19418 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 07:01:04.319753 19418 webserver.cc:460] Webserver started at http://127.1.74.65:34263/ using document root <none> and password file <none>
I20260926 07:01:04.320000 19418 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:04.320070 19418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:04.320204 19418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:04.321254 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "57e483957e264b4d8236b67031bffa88"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "ffffa90f8ef0a33b847b8b6407df3d45"
server_key_iv: "cd39ba4718fe8a194e0223ade5aeb5b7"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.321621 19418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "57e483957e264b4d8236b67031bffa88"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "ffffa90f8ef0a33b847b8b6407df3d45"
server_key_iv: "cd39ba4718fe8a194e0223ade5aeb5b7"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.322926 19418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:04.323688 19433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.323863 19418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.323961 19418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "57e483957e264b4d8236b67031bffa88"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "ffffa90f8ef0a33b847b8b6407df3d45"
server_key_iv: "cd39ba4718fe8a194e0223ade5aeb5b7"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.324043 19418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:04.335678 19418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:04.337566 19418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:04.337730 19418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:04.337989 19418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:04.338306 19418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:04.338367 19418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.338411 19418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:04.338436 19418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.344465 19418 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:44857
I20260926 07:01:04.344506 19546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:44857 every 8 connection(s)
I20260926 07:01:04.344866 19418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 07:01:04.350575 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 19418
I20260926 07:01:04.350692 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:04.350916 1321 external_mini_cluster.cc:1428] Setting key d5d58325a4da8911ae51a14e2df5176f
Sep 26 07:01:04 dist-test-slave-sfvt krb5kdc[19311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:04.354511 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.345607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:38999 (local address 127.1.74.126:42399)
0926 07:01:04.345688 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.345691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.345874 (+ 183us) server_negotiation.cc:407] Connection header received
0926 07:01:04.346337 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.346338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.346349 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.346397 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:04.347370 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.347966 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.349003 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.349209 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.349266 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.351724 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:04.351726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:04.351726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:04.351733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:04.353053 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.353494 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.353496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.353496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.353559 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.353822 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.353823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.353824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.353945 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:04.354018 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.354322 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.354416 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:01:04.354811 19549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.345549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42399 (local address 127.1.74.65:38999)
0926 07:01:04.345709 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:04.345747 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:04.346226 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:04.346461 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:04.346465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:04.347031 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:04.347230 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.347234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.348043 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.348044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.348928 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.348930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.349287 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.349287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.349386 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.349940 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:04.349949 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:04.351620 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:04.353156 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:04.353163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:04.353165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:04.353401 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:04.353627 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:04.353627 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:04.353628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:04.353747 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:04.354018 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:04.354026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:04.354172 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 07:01:04.354284 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:01:04.354451 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260926 07:01:04.355386 19547 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42399
I20260926 07:01:04.355484 19547 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:04.355690 19547 heartbeater.cc:511] Master 127.1.74.126:42399 requested a full tablet report, sending...
I20260926 07:01:04.356216 19356 ts_manager.cc:194] Registered new tserver with Master: 57e483957e264b4d8236b67031bffa88 (127.1.74.65:44857)
I20260926 07:01:04.356848 19356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:38999
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 07:01:04.379209 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42399
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:04.461344 19552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:04.461620 19552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:04.461668 19552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:04.462847 19552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:04.462926 19552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:04.462976 19552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 07:01:04.464507 19552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:04.464808 19552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:04.465063 19552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:04.470118 19557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:04.470126 19558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:04.470129 19560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:04.470487 19552 server_base.cc:1034] running on GCE node
I20260926 07:01:04.470700 19552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:04.470906 19552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:04.472076 19552 hybrid_clock.cc:648] HybridClock initialized: now 1790406064472048 us; error 45 us; skew 500 ppm
Sep 26 07:01:04 dist-test-slave-sfvt krb5kdc[19311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:04.474748 19552 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 07:01:04.475186 19552 webserver.cc:460] Webserver started at http://127.1.74.66:35167/ using document root <none> and password file <none>
I20260926 07:01:04.475428 19552 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:04.475497 19552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:04.475646 19552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:04.476680 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "5fef0046488c4509b171cdfe1e7ce6b9"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "779c8f60a159dac2fe9522ed2be5f37a"
server_key_iv: "affd9e807f4ec9a9e6ef5ec83e49e720"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.477046 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "5fef0046488c4509b171cdfe1e7ce6b9"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "779c8f60a159dac2fe9522ed2be5f37a"
server_key_iv: "affd9e807f4ec9a9e6ef5ec83e49e720"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.478343 19552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:04.479050 19567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.479223 19552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:04.479355 19552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "5fef0046488c4509b171cdfe1e7ce6b9"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "779c8f60a159dac2fe9522ed2be5f37a"
server_key_iv: "affd9e807f4ec9a9e6ef5ec83e49e720"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.479430 19552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:04.487581 19552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:04.489367 19552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:04.489538 19552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:04.489758 19552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:04.490072 19552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:04.490106 19552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.490149 19552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:04.490185 19552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.495956 19552 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:45727
I20260926 07:01:04.496006 19680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:45727 every 8 connection(s)
I20260926 07:01:04.496304 19552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:01:04 dist-test-slave-sfvt krb5kdc[19311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:04.503913 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 19552
I20260926 07:01:04.504029 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:04.504248 1321 external_mini_cluster.cc:1428] Setting key 5db6a54a8b73f0e8d4bf08c701cfd950
I20260926 07:01:04.505188 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.496994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:40609 (local address 127.1.74.126:42399)
0926 07:01:04.497071 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.497073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.497224 (+ 151us) server_negotiation.cc:407] Connection header received
0926 07:01:04.497678 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.497679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.497689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.497741 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:04.498618 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.499096 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.500105 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.500294 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.500349 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.502379 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:04.502380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:04.502380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:04.502387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:04.503661 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.504172 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.504172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.504173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.504223 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.504520 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.504521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.504522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.504673 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:04.504740 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.505023 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.505110 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:01:04.505414 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.496930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42399 (local address 127.1.74.66:40609)
0926 07:01:04.497097 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:04.497132 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:04.497599 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:04.497808 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:04.497812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:04.498357 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:04.498543 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.498547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.499175 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.499175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.500032 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.500034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.500367 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.500367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.500462 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.500908 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:04.500916 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:04.502289 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:04.503772 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:04.503779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:04.503782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:04.504098 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:04.504298 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:04.504299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:04.504299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:04.504444 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:04.504739 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:04.504744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:04.504860 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:01:04.504988 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 07:01:04.505157 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260926 07:01:04.505875 19681 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42399
I20260926 07:01:04.505971 19681 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:04.506166 19681 heartbeater.cc:511] Master 127.1.74.126:42399 requested a full tablet report, sending...
I20260926 07:01:04.506554 19356 ts_manager.cc:194] Registered new tserver with Master: 5fef0046488c4509b171cdfe1e7ce6b9 (127.1.74.66:45727)
I20260926 07:01:04.507082 19356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:40609
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 07:01:04.530287 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42399
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:04.610967 19686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:04.611176 19686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:04.611199 19686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:04.612390 19686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:04.612476 19686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:04.612524 19686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 07:01:04.613916 19686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:04.614096 19686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:04.614283 19686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:04.619195 19692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:04.619236 19691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:04.619196 19694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:04.619508 19686 server_base.cc:1034] running on GCE node
I20260926 07:01:04.619736 19686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:04.619971 19686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:04.621109 19686 hybrid_clock.cc:648] HybridClock initialized: now 1790406064621100 us; error 31 us; skew 500 ppm
Sep 26 07:01:04 dist-test-slave-sfvt krb5kdc[19311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:04.623598 19686 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 07:01:04.624033 19686 webserver.cc:460] Webserver started at http://127.1.74.67:35837/ using document root <none> and password file <none>
I20260926 07:01:04.624259 19686 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:04.624305 19686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:04.624457 19686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:04.625527 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "4d45abd559764b4ca29fe4e776c9c750"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "fe1de7984f7d6384d26fb0a411f867a1"
server_key_iv: "0f799168ea8f62b46ec6b14f79fcff70"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.625891 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "4d45abd559764b4ca29fe4e776c9c750"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "fe1de7984f7d6384d26fb0a411f867a1"
server_key_iv: "0f799168ea8f62b46ec6b14f79fcff70"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.627385 19686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:04.628181 19701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.628362 19686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:04.628453 19686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "4d45abd559764b4ca29fe4e776c9c750"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "fe1de7984f7d6384d26fb0a411f867a1"
server_key_iv: "0f799168ea8f62b46ec6b14f79fcff70"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.628527 19686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:04.643016 19686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:04.645030 19686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:04.645177 19686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:04.645442 19686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:04.645758 19686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:04.645792 19686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.645814 19686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:04.645828 19686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.652495 19686 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:46307
I20260926 07:01:04.652537 19814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:46307 every 8 connection(s)
I20260926 07:01:04.652881 19686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 07:01:04.655092 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 19686
I20260926 07:01:04.655217 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:04.655431 1321 external_mini_cluster.cc:1428] Setting key d437cdb2655749aef8459a8e3bd24d8b
Sep 26 07:01:04 dist-test-slave-sfvt krb5kdc[19311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:04.662694 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.653609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:56475 (local address 127.1.74.126:42399)
0926 07:01:04.653657 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.653660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.653915 (+ 255us) server_negotiation.cc:407] Connection header received
0926 07:01:04.654454 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.654455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.654466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.654522 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:04.655599 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.656321 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.657303 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.657509 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.657584 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.659865 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:04.659867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:04.659867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:04.659876 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:04.661258 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.661712 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.661713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.661714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.661780 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.662032 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.662033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.662034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.662162 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:04.662243 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.662513 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.662599 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:04.662899 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.653614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42399 (local address 127.1.74.67:56475)
0926 07:01:04.653791 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:04.653825 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:04.654346 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:04.654594 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:04.654598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:04.655171 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:04.655469 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.655474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.656394 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.656395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.657203 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.657205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.657590 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.657591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.657732 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.658235 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:04.658243 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:04.659738 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:04.661372 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:04.661380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:04.661382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:04.661617 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:04.661848 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:04.661849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:04.661849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:04.661965 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:04.662234 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:04.662240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:04.662384 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 07:01:04.662495 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:01:04.662657 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260926 07:01:04.663400 19815 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42399
I20260926 07:01:04.663497 19815 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:04.663648 19815 heartbeater.cc:511] Master 127.1.74.126:42399 requested a full tablet report, sending...
I20260926 07:01:04.664052 19356 ts_manager.cc:194] Registered new tserver with Master: 4d45abd559764b4ca29fe4e776c9c750 (127.1.74.67:46307)
I20260926 07:01:04.664391 19356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:56475
I20260926 07:01:04.667356 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:01:04.675272 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.668694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.1.74.126:42399)
0926 07:01:04.668750 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.668754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.668864 (+ 110us) server_negotiation.cc:407] Connection header received
0926 07:01:04.668940 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.668941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.668951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.669004 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:04.669564 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.670035 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.670784 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.670987 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.671070 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.672442 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:04.672443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:04.672444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:04.672451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:04.674171 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.674556 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.674557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.674558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.674614 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:04.674801 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:04.674801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:04.674801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:04.674922 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:04.675002 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.675102 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.675151 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:01:04.676913 19356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:04.678459 19356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:04.684702 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.680871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.1.74.65:44857)
0926 07:01:04.681064 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.681068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.681102 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:01:04.681186 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.681188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.681203 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.681283 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:04.681390 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.682570 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.683814 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.684306 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.684404 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.684467 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.684582 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.684620 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260926 07:01:04.684702 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.680712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44857 (local address 127.0.0.1:33702)
0926 07:01:04.681050 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:04.681059 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:04.681120 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:04.681293 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:04.681294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:04.681295 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:04.681331 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.681332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.682680 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.682681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.683729 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.683731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.684406 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.684407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.684482 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.684495 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:01:04.684567 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 07:01:04.684619 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":37,"threads_started":1}
I20260926 07:01:04.685065 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.680629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.1.74.67:46307)
0926 07:01:04.680810 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.680814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.680933 (+ 119us) server_negotiation.cc:407] Connection header received
0926 07:01:04.680980 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.680983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.680999 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.681077 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:04.681356 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.682350 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.683863 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.684563 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.684635 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.684686 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.684947 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.684991 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260926 07:01:04.685005 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.680502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:45727 (local address 127.0.0.1:38226)
0926 07:01:04.680662 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:04.680716 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:04.680783 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:04.681160 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:04.681163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:04.681165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:04.681261 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.681263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.682643 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.682643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.683761 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.683764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.684724 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.684725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.684810 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.684815 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:04.684873 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:01:04.684922 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":27,"thread_start_us":54,"threads_started":1}
I20260926 07:01:04.685015 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.680536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:46307 (local address 127.0.0.1:45678)
0926 07:01:04.680800 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:04.680811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:04.680867 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:04.681161 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:04.681163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:04.681166 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:04.681274 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.681277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.682446 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.682447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.683768 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.683769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.684725 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.684726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.684860 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.684863 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:04.684915 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:01:04.684942 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260926 07:01:04.685207 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.680606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.1.74.66:45727)
0926 07:01:04.680894 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.680898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.680907 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:04.680954 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.680956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.680989 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.681068 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:04.681407 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.682533 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.683863 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.684563 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.684641 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.684687 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.685081 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.685125 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":77,"threads_started":1}
I20260926 07:01:04.685983 19615 tablet_service.cc:1467] Processing CreateTablet for tablet 723b7eb718d24a1ead3566973a48abd1 (DEFAULT_TABLE table=test-table [id=f7754f3b88d64546b211414d74a4ea46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:04.686028 19481 tablet_service.cc:1467] Processing CreateTablet for tablet 723b7eb718d24a1ead3566973a48abd1 (DEFAULT_TABLE table=test-table [id=f7754f3b88d64546b211414d74a4ea46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:04.686296 19481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723b7eb718d24a1ead3566973a48abd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:04.686303 19615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723b7eb718d24a1ead3566973a48abd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:04.687877 19749 tablet_service.cc:1467] Processing CreateTablet for tablet 723b7eb718d24a1ead3566973a48abd1 (DEFAULT_TABLE table=test-table [id=f7754f3b88d64546b211414d74a4ea46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:04.688185 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723b7eb718d24a1ead3566973a48abd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:04.689216 19829 tablet_bootstrap.cc:492] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: Bootstrap starting.
I20260926 07:01:04.689837 19829 tablet_bootstrap.cc:654] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:04.690328 19829 log.cc:824] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:04.691052 19829 tablet_bootstrap.cc:492] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: No bootstrap required, opened a new log
I20260926 07:01:04.691030 19832 tablet_bootstrap.cc:492] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: Bootstrap starting.
I20260926 07:01:04.691123 19829 ts_tablet_manager.cc:1397] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:04.691115 19830 tablet_bootstrap.cc:492] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88: Bootstrap starting.
I20260926 07:01:04.691665 19830 tablet_bootstrap.cc:654] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:04.691792 19832 tablet_bootstrap.cc:654] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:04.692281 19830 log.cc:824] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:04.692407 19832 log.cc:824] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:04.693079 19830 tablet_bootstrap.cc:492] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88: No bootstrap required, opened a new log
I20260926 07:01:04.693156 19830 ts_tablet_manager.cc:1397] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:04.693456 19829 raft_consensus.cc:348] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } }
I20260926 07:01:04.693635 19829 raft_consensus.cc:374] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:04.693667 19829 raft_consensus.cc:729] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fef0046488c4509b171cdfe1e7ce6b9, State: Initialized, Role: FOLLOWER
I20260926 07:01:04.693789 19829 consensus_queue.cc:260] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } }
I20260926 07:01:04.694056 19829 ts_tablet_manager.cc:1428] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:04.694084 19681 heartbeater.cc:503] Master 127.1.74.126:42399 was elected leader, sending a full tablet report...
I20260926 07:01:04.695037 19830 raft_consensus.cc:348] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } }
I20260926 07:01:04.695191 19830 raft_consensus.cc:374] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:04.695232 19830 raft_consensus.cc:729] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e483957e264b4d8236b67031bffa88, State: Initialized, Role: FOLLOWER
I20260926 07:01:04.695369 19830 consensus_queue.cc:260] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } }
I20260926 07:01:04.695624 19830 ts_tablet_manager.cc:1428] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 07:01:04.695653 19547 heartbeater.cc:503] Master 127.1.74.126:42399 was elected leader, sending a full tablet report...
I20260926 07:01:04.696069 19832 tablet_bootstrap.cc:492] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: No bootstrap required, opened a new log
I20260926 07:01:04.696144 19832 ts_tablet_manager.cc:1397] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 07:01:04.697574 19832 raft_consensus.cc:348] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } }
I20260926 07:01:04.697718 19832 raft_consensus.cc:374] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:04.697742 19832 raft_consensus.cc:729] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d45abd559764b4ca29fe4e776c9c750, State: Initialized, Role: FOLLOWER
I20260926 07:01:04.697842 19832 consensus_queue.cc:260] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } }
I20260926 07:01:04.698056 19832 ts_tablet_manager.cc:1428] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:04.698110 19815 heartbeater.cc:503] Master 127.1.74.126:42399 was elected leader, sending a full tablet report...
I20260926 07:01:04.741379 19837 raft_consensus.cc:482] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:04.741494 19837 raft_consensus.cc:504] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } }
I20260926 07:01:04.741922 19837 leader_election.cc:288] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fef0046488c4509b171cdfe1e7ce6b9 (127.1.74.66:45727), 57e483957e264b4d8236b67031bffa88 (127.1.74.65:44857)
I20260926 07:01:04.745782 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.742343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:49873 (local address 127.1.74.65:44857)
0926 07:01:04.742394 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.742396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.742572 (+ 176us) server_negotiation.cc:407] Connection header received
0926 07:01:04.742600 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.742601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.742614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.742663 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:04.742803 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.743501 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.744802 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.745387 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.745465 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.745476 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.745668 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.745710 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 07:01:04.745782 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.742173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:45441 (local address 127.1.74.66:45727)
0926 07:01:04.742256 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.742258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.742268 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:04.742337 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.742338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.742350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.742411 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:04.742571 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.743388 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.744799 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.745387 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.745465 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.745476 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.745669 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.745710 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:01:04.745779 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.742125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:45727 (local address 127.1.74.67:45441)
0926 07:01:04.742172 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:04.742183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:04.742264 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:04.742418 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:04.742419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:04.742422 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:04.742498 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.742501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.743468 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.743468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.744689 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.744691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.745476 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.745476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.745591 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.745594 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:04.745645 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 07:01:04.745690 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":35}
I20260926 07:01:04.745790 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.742269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44857 (local address 127.1.74.67:49873)
0926 07:01:04.742458 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:04.742468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:04.742529 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:04.742675 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:04.742676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:04.742677 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:04.742730 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.742731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.743583 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.743583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.744689 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:04.744691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.745476 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.745477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:04.745572 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.745577 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:04.745638 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:01:04.745683 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260926 07:01:04.746168 19635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723b7eb718d24a1ead3566973a48abd1" candidate_uuid: "4d45abd559764b4ca29fe4e776c9c750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" is_pre_election: true
I20260926 07:01:04.746258 19501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723b7eb718d24a1ead3566973a48abd1" candidate_uuid: "4d45abd559764b4ca29fe4e776c9c750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e483957e264b4d8236b67031bffa88" is_pre_election: true
I20260926 07:01:04.746337 19635 raft_consensus.cc:2454] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d45abd559764b4ca29fe4e776c9c750 in term 0.
I20260926 07:01:04.746426 19501 raft_consensus.cc:2454] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d45abd559764b4ca29fe4e776c9c750 in term 0.
I20260926 07:01:04.746554 19703 leader_election.cc:302] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d45abd559764b4ca29fe4e776c9c750, 5fef0046488c4509b171cdfe1e7ce6b9; no voters:
I20260926 07:01:04.746694 19837 raft_consensus.cc:2781] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:04.746742 19837 raft_consensus.cc:482] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:04.746807 19837 raft_consensus.cc:3037] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 0 FOLLOWER]: Advancing to term 1
W20260926 07:01:04.746918 19682 tablet.cc:2369] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:01:04.747617 19837 raft_consensus.cc:504] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } }
I20260926 07:01:04.747751 19837 leader_election.cc:288] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [CANDIDATE]: Term 1 election: Requested vote from peers 5fef0046488c4509b171cdfe1e7ce6b9 (127.1.74.66:45727), 57e483957e264b4d8236b67031bffa88 (127.1.74.65:44857)
I20260926 07:01:04.747912 19635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723b7eb718d24a1ead3566973a48abd1" candidate_uuid: "4d45abd559764b4ca29fe4e776c9c750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fef0046488c4509b171cdfe1e7ce6b9"
I20260926 07:01:04.747982 19501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723b7eb718d24a1ead3566973a48abd1" candidate_uuid: "4d45abd559764b4ca29fe4e776c9c750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e483957e264b4d8236b67031bffa88"
I20260926 07:01:04.747999 19635 raft_consensus.cc:3037] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:04.748064 19501 raft_consensus.cc:3037] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:04.748911 19635 raft_consensus.cc:2454] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d45abd559764b4ca29fe4e776c9c750 in term 1.
I20260926 07:01:04.748982 19501 raft_consensus.cc:2454] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d45abd559764b4ca29fe4e776c9c750 in term 1.
I20260926 07:01:04.749101 19703 leader_election.cc:302] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d45abd559764b4ca29fe4e776c9c750, 5fef0046488c4509b171cdfe1e7ce6b9; no voters:
I20260926 07:01:04.749207 19837 raft_consensus.cc:2781] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:04.749383 19837 raft_consensus.cc:686] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 1 LEADER]: Becoming Leader. State: Replica: 4d45abd559764b4ca29fe4e776c9c750, State: Running, Role: LEADER
I20260926 07:01:04.749511 19837 consensus_queue.cc:237] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } }
I20260926 07:01:04.750473 19356 catalog_manager.cc:5495] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d45abd559764b4ca29fe4e776c9c750 (127.1.74.67). New cstate: current_term: 1 leader_uuid: "4d45abd559764b4ca29fe4e776c9c750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d45abd559764b4ca29fe4e776c9c750" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 46307 } health_report { overall_health: HEALTHY } } }
I20260926 07:01:04.757287 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.754870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.1.74.67:46307)
0926 07:01:04.754913 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.754915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.754923 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:04.754989 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.754990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.755015 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.755092 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:04.755247 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.755727 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.756509 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.756704 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.756791 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.756956 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:04.757045 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:04.757153 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.757183 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.757192 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:01:04.758863 19635 raft_consensus.cc:1261] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [term 1 FOLLOWER]: Refusing update from remote peer 4d45abd559764b4ca29fe4e776c9c750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:04.758860 19501 raft_consensus.cc:1261] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88 [term 1 FOLLOWER]: Refusing update from remote peer 4d45abd559764b4ca29fe4e776c9c750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:04.759178 19837 consensus_queue.cc:1035] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fef0046488c4509b171cdfe1e7ce6b9" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:04.759346 19839 consensus_queue.cc:1035] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e483957e264b4d8236b67031bffa88" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 44857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:04.761730 19843 mvcc.cc:205] Tried to move back new op lower bound from 7333503241250369536 to 7333503241215225856. Current Snapshot: MvccSnapshot[applied={T|T < 7333503241250369536}]
I20260926 07:01:04.763372 19355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:04.763466 19355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:04.764196 19355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec8f9d0634674863a214cd76257f35f1: Sending DeleteTablet for 3 replicas of tablet 723b7eb718d24a1ead3566973a48abd1
I20260926 07:01:04.764505 19615 tablet_service.cc:1514] Processing DeleteTablet for tablet 723b7eb718d24a1ead3566973a48abd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:04 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:38226
I20260926 07:01:04.764503 19749 tablet_service.cc:1514] Processing DeleteTablet for tablet 723b7eb718d24a1ead3566973a48abd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:04 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:45678
I20260926 07:01:04.764557 19481 tablet_service.cc:1514] Processing DeleteTablet for tablet 723b7eb718d24a1ead3566973a48abd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:04 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:33702
I20260926 07:01:04.764734 19853 tablet_replica.cc:330] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: stopping tablet replica
I20260926 07:01:04.764837 19853 raft_consensus.cc:2229] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:01:04.764925 19853 raft_consensus.cc:2258] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:04.764966 19852 tablet_replica.cc:330] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: stopping tablet replica
I20260926 07:01:04.765023 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 19418
I20260926 07:01:04.765057 19852 raft_consensus.cc:2229] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:04.765058 19854 tablet_replica.cc:330] T 723b7eb718d24a1ead3566973a48abd1 P 57e483957e264b4d8236b67031bffa88: stopping tablet replica
I20260926 07:01:04.765151 19852 raft_consensus.cc:2258] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:04.765974 19853 ts_tablet_manager.cc:1905] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:04.767192 19852 ts_tablet_manager.cc:1905] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:04.768705 19852 ts_tablet_manager.cc:1918] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:04.768795 19852 log.cc:1196] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/723b7eb718d24a1ead3566973a48abd1
I20260926 07:01:04.769037 19852 ts_tablet_manager.cc:1939] T 723b7eb718d24a1ead3566973a48abd1 P 5fef0046488c4509b171cdfe1e7ce6b9: Deleting consensus metadata
I20260926 07:01:04.769815 19341 catalog_manager.cc:4841] TS 5fef0046488c4509b171cdfe1e7ce6b9 (127.1.74.66:45727): tablet 723b7eb718d24a1ead3566973a48abd1 (table test-table [id=f7754f3b88d64546b211414d74a4ea46]) successfully deleted
I20260926 07:01:04.769882 19853 ts_tablet_manager.cc:1918] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:04.769956 19853 log.cc:1196] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/723b7eb718d24a1ead3566973a48abd1
I20260926 07:01:04.770233 19853 ts_tablet_manager.cc:1939] T 723b7eb718d24a1ead3566973a48abd1 P 4d45abd559764b4ca29fe4e776c9c750: Deleting consensus metadata
I20260926 07:01:04.770612 19343 catalog_manager.cc:4841] TS 4d45abd559764b4ca29fe4e776c9c750 (127.1.74.67:46307): tablet 723b7eb718d24a1ead3566973a48abd1 (table test-table [id=f7754f3b88d64546b211414d74a4ea46]) successfully deleted
W20260926 07:01:04.772282 19341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:44857 (error 108)
I20260926 07:01:04.772673 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 19552
I20260926 07:01:04.772746 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.772586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:44857 (local address 127.0.0.1:33716)
0926 07:01:04.772665 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:04.772712 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 07:01:04.772857 19341 catalog_manager.cc:4584] TS 57e483957e264b4d8236b67031bffa88 (127.1.74.65:44857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 723b7eb718d24a1ead3566973a48abd1: Network error: Client connection negotiation failed: client connection to 127.1.74.65:44857: connect: Connection refused (error 111)
I20260926 07:01:04.777361 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 19686
I20260926 07:01:04.784287 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 19324
2026-09-26T07:01:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1892 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T07:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:04Z Disabled control of system clock
I20260926 07:01:04.811875 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42779
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:42779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:01:04.891494 19859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:04.891716 19859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:04.891736 19859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:04.892812 19859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:04.892848 19859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:04.892860 19859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:04.892874 19859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:04.892885 19859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:04.894143 19859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:42779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:42779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:04.894387 19859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:04.894767 19859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:04.900081 19864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:04.900097 19865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:04.900113 19867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:04.901089 19859 server_base.cc:1034] running on GCE node
I20260926 07:01:04.901412 19859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:04.901782 19859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:04.902922 19859 hybrid_clock.cc:648] HybridClock initialized: now 1790406064902911 us; error 27 us; skew 500 ppm
I20260926 07:01:04.904595 19859 webserver.cc:460] Webserver started at http://127.1.74.126:44189/ using document root <none> and password file <none>
I20260926 07:01:04.904892 19859 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:04.904979 19859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:04.905124 19859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:04.906219 19859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "2736944fe7ad4b248eee9dfd92ffa42e"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "cb3d094f38a5ac65e266c4babfabf80b"
server_key_iv: "81020f5f100f7b519cb46e1e3f3ea837"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.906582 19859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "2736944fe7ad4b248eee9dfd92ffa42e"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "cb3d094f38a5ac65e266c4babfabf80b"
server_key_iv: "81020f5f100f7b519cb46e1e3f3ea837"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.907979 19859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 07:01:04.908684 19873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:04.908859 19859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:04.908965 19859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "2736944fe7ad4b248eee9dfd92ffa42e"
format_stamp: "Formatted at 2026-09-26 07:01:04 on dist-test-slave-sfvt"
server_key: "cb3d094f38a5ac65e266c4babfabf80b"
server_key_iv: "81020f5f100f7b519cb46e1e3f3ea837"
server_key_version: "encryptionkey@0"
I20260926 07:01:04.909050 19859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:04.921624 19859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:04.921958 19859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:04.922122 19859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:04.926277 19859 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:42779
I20260926 07:01:04.926323 19925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:42779 every 8 connection(s)
I20260926 07:01:04.926692 19859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:04.927738 19926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:04.930037 19926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e: Bootstrap starting.
I20260926 07:01:04.930624 19926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:04.930924 19926 log.cc:824] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:04.931600 19926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e: No bootstrap required, opened a new log
I20260926 07:01:04.933178 19926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2736944fe7ad4b248eee9dfd92ffa42e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42779 } }
I20260926 07:01:04.933312 19926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:04.933336 19926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2736944fe7ad4b248eee9dfd92ffa42e, State: Initialized, Role: FOLLOWER
I20260926 07:01:04.933514 19926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2736944fe7ad4b248eee9dfd92ffa42e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42779 } }
I20260926 07:01:04.933580 19926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:04.933622 19926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:04.933670 19926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:04.934170 19926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2736944fe7ad4b248eee9dfd92ffa42e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42779 } }
I20260926 07:01:04.934288 19926 leader_election.cc:302] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2736944fe7ad4b248eee9dfd92ffa42e; no voters:
I20260926 07:01:04.934475 19926 leader_election.cc:288] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:04.934576 19929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:04.934757 19926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:04.934785 19929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [term 1 LEADER]: Becoming Leader. State: Replica: 2736944fe7ad4b248eee9dfd92ffa42e, State: Running, Role: LEADER
I20260926 07:01:04.934897 19929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2736944fe7ad4b248eee9dfd92ffa42e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42779 } }
I20260926 07:01:04.935339 19931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2736944fe7ad4b248eee9dfd92ffa42e. Latest consensus state: current_term: 1 leader_uuid: "2736944fe7ad4b248eee9dfd92ffa42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2736944fe7ad4b248eee9dfd92ffa42e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42779 } } }
I20260926 07:01:04.935423 19930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2736944fe7ad4b248eee9dfd92ffa42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2736944fe7ad4b248eee9dfd92ffa42e" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 42779 } } }
I20260926 07:01:04.935441 19931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:04.935472 19930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:04.935969 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 19859
I20260926 07:01:04.936084 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:04.936126 19938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:04.936301 1321 external_mini_cluster.cc:1428] Setting key e1172365128f864fc84cee909581d221
I20260926 07:01:04.936604 19938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:04.938030 19938 catalog_manager.cc:1347] Generated new cluster ID: 6c05235027944dd3bb429666bb5d91ad
I20260926 07:01:04.938097 19938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:04.941665 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:04.936786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.1.74.126:42779)
0926 07:01:04.936984 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:04.936991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:04.937159 (+ 168us) server_negotiation.cc:407] Connection header received
0926 07:01:04.937306 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:04.937312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:04.937365 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:04.937506 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:04.937736 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.938725 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.939662 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:04.940089 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:04.940228 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:04.940923 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:04.940927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:04.940927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:04.941122 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:04.941154 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:04.941165 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:04.941237 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:04.941319 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 07:01:04.941497 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260926 07:01:04.952203 19938 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:04.952889 19938 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:04.962317 19938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e: Generated new TSK 0
I20260926 07:01:04.962535 19938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:04.966562 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42779
--builtin_ntp_servers=127.1.74.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:01:05.055891 19950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:05.056093 19950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:05.056115 19950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:05.057273 19950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:05.057313 19950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:05.057337 19950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 07:01:05.058713 19950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:05.058954 19950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:05.059295 19950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:05.065052 19956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:05.065119 19955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:05.065217 19950 server_base.cc:1034] running on GCE node
W20260926 07:01:05.065397 19958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:05.065623 19950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:05.065897 19950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:05.067046 19950 hybrid_clock.cc:648] HybridClock initialized: now 1790406065067028 us; error 34 us; skew 500 ppm
I20260926 07:01:05.068394 19950 webserver.cc:460] Webserver started at http://127.1.74.65:39383/ using document root <none> and password file <none>
I20260926 07:01:05.068660 19950 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:05.068729 19950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:05.068876 19950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:05.069976 19950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "aca4e9b31187457496f7941388db35b0"
format_stamp: "Formatted at 2026-09-26 07:01:05 on dist-test-slave-sfvt"
server_key: "d456ef7e12ec490494e6acb04c6bfb53"
server_key_iv: "f9bda4d9485bfef45d7a053024eab92d"
server_key_version: "encryptionkey@0"
I20260926 07:01:05.070333 19950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "aca4e9b31187457496f7941388db35b0"
format_stamp: "Formatted at 2026-09-26 07:01:05 on dist-test-slave-sfvt"
server_key: "d456ef7e12ec490494e6acb04c6bfb53"
server_key_iv: "f9bda4d9485bfef45d7a053024eab92d"
server_key_version: "encryptionkey@0"
I20260926 07:01:05.071732 19950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:05.072520 19964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.072691 19950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:05.072795 19950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "aca4e9b31187457496f7941388db35b0"
format_stamp: "Formatted at 2026-09-26 07:01:05 on dist-test-slave-sfvt"
server_key: "d456ef7e12ec490494e6acb04c6bfb53"
server_key_iv: "f9bda4d9485bfef45d7a053024eab92d"
server_key_version: "encryptionkey@0"
I20260926 07:01:05.072867 19950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:05.082656 19950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:05.082968 19950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:05.083120 19950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:05.083406 19950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:05.083739 19950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:05.083771 19950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.083792 19950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:05.083838 19950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.089433 19950 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:35381
I20260926 07:01:05.089497 20077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:35381 every 8 connection(s)
I20260926 07:01:05.089808 19950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 07:01:05.091831 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 19950
I20260926 07:01:05.091919 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:05.092104 1321 external_mini_cluster.cc:1428] Setting key fe7cc55438c6632ebecc869a6641d179
I20260926 07:01:05.093015 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42779
--builtin_ntp_servers=127.1.74.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:01:05.095466 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.090532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:52533 (local address 127.1.74.126:42779)
0926 07:01:05.090605 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.090608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.090854 (+ 246us) server_negotiation.cc:407] Connection header received
0926 07:01:05.091348 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.091349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.091368 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.091424 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:05.092172 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.092711 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.093806 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.094152 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.094227 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.095132 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:05.095133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:05.095133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:05.095138 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:05.095152 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:05.095156 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:05.095225 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.095348 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.095378 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:01:05.095675 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.090524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42779 (local address 127.1.74.65:52533)
0926 07:01:05.090732 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:05.090765 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:05.091243 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:05.091502 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:05.091506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:05.091515 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:05.092079 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.092084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.092823 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.092824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.093719 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.093722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.094295 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.094296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.094409 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.094974 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:01:05.094974 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:05.095003 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:01:05.095025 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:05.095226 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:05.095232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:05.095235 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:05.095349 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:01:05.095453 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260926 07:01:05.096124 20078 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42779
I20260926 07:01:05.096233 20078 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:05.096458 20078 heartbeater.cc:511] Master 127.1.74.126:42779 requested a full tablet report, sending...
I20260926 07:01:05.096956 19890 ts_manager.cc:194] Registered new tserver with Master: aca4e9b31187457496f7941388db35b0 (127.1.74.65:35381)
I20260926 07:01:05.097592 19890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.65:52533
W20260926 07:01:05.176131 20081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:05.176321 20081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:05.176342 20081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:05.177474 20081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:05.177512 20081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:05.177536 20081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 07:01:05.178784 20081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:05.179041 20081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:05.179375 20081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:05.184585 20089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:05.184599 20086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:05.184649 20081 server_base.cc:1034] running on GCE node
W20260926 07:01:05.184593 20087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:05.185032 20081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:05.185300 20081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:05.186439 20081 hybrid_clock.cc:648] HybridClock initialized: now 1790406065186417 us; error 37 us; skew 500 ppm
I20260926 07:01:05.187666 20081 webserver.cc:460] Webserver started at http://127.1.74.66:41367/ using document root <none> and password file <none>
I20260926 07:01:05.187911 20081 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:05.187958 20081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:05.188122 20081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:05.189186 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "969c4eba644e4e1fb47d76103251aac5"
format_stamp: "Formatted at 2026-09-26 07:01:05 on dist-test-slave-sfvt"
server_key: "67764bafef9d556e88d509e1fe2a5224"
server_key_iv: "d3c8aa6a6daf63fd0c5612593e106d8b"
server_key_version: "encryptionkey@0"
I20260926 07:01:05.189543 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "969c4eba644e4e1fb47d76103251aac5"
format_stamp: "Formatted at 2026-09-26 07:01:05 on dist-test-slave-sfvt"
server_key: "67764bafef9d556e88d509e1fe2a5224"
server_key_iv: "d3c8aa6a6daf63fd0c5612593e106d8b"
server_key_version: "encryptionkey@0"
I20260926 07:01:05.190841 20081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:05.191584 20095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.191745 20081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.191815 20081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "969c4eba644e4e1fb47d76103251aac5"
format_stamp: "Formatted at 2026-09-26 07:01:05 on dist-test-slave-sfvt"
server_key: "67764bafef9d556e88d509e1fe2a5224"
server_key_iv: "d3c8aa6a6daf63fd0c5612593e106d8b"
server_key_version: "encryptionkey@0"
I20260926 07:01:05.191907 20081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:05.212724 20081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:05.213059 20081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:05.213212 20081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:05.213454 20081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:05.213804 20081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:05.213860 20081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.213894 20081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:05.213924 20081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.219834 20081 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:44125
I20260926 07:01:05.219887 20208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:44125 every 8 connection(s)
I20260926 07:01:05.220197 20081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 07:01:05.225131 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.220904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:59337 (local address 127.1.74.126:42779)
0926 07:01:05.220987 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.220989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.221192 (+ 203us) server_negotiation.cc:407] Connection header received
0926 07:01:05.221662 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.221663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.221673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.221727 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:05.222380 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.222864 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.223842 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.224047 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.224101 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.224925 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:05.224926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:05.224926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:05.224930 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:05.224940 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:05.224944 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:05.224981 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.225053 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.225072 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:01:05.225378 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.220886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42779 (local address 127.1.74.66:59337)
0926 07:01:05.221077 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:05.221111 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:05.221577 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:05.221779 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:05.221783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:05.221790 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:05.222298 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.222302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.222938 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.222939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.223774 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.223776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.224117 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.224117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.224221 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.224775 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:01:05.224775 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:05.224812 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:01:05.224854 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:05.224992 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:05.224995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:05.224997 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:05.225044 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 07:01:05.225118 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260926 07:01:05.225878 20209 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42779
I20260926 07:01:05.225994 20209 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:05.226172 20209 heartbeater.cc:511] Master 127.1.74.126:42779 requested a full tablet report, sending...
I20260926 07:01:05.226549 19890 ts_manager.cc:194] Registered new tserver with Master: 969c4eba644e4e1fb47d76103251aac5 (127.1.74.66:44125)
I20260926 07:01:05.226886 19890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.66:59337
I20260926 07:01:05.228142 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 20081
I20260926 07:01:05.228235 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:05.228392 1321 external_mini_cluster.cc:1428] Setting key 4d5c6185c5b77f44a2ff23cbd400780e
I20260926 07:01:05.229225 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42779
--builtin_ntp_servers=127.1.74.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:01:05.309423 20212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:05.309659 20212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:05.309700 20212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:05.310856 20212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:05.310921 20212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:05.310963 20212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 07:01:05.312485 20212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:42779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:05.312754 20212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:05.313060 20212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:05.318025 20218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:05.318200 20212 server_base.cc:1034] running on GCE node
W20260926 07:01:05.318032 20217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:05.318034 20220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:05.318725 20212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:05.319017 20212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:05.320145 20212 hybrid_clock.cc:648] HybridClock initialized: now 1790406065320089 us; error 71 us; skew 500 ppm
I20260926 07:01:05.321426 20212 webserver.cc:460] Webserver started at http://127.1.74.67:41603/ using document root <none> and password file <none>
I20260926 07:01:05.321691 20212 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:05.321738 20212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:05.321888 20212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:05.322971 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1"
format_stamp: "Formatted at 2026-09-26 07:01:05 on dist-test-slave-sfvt"
server_key: "6e1fad34f292b5349032714908476079"
server_key_iv: "bd043f419c8a3023363c1a5de5155855"
server_key_version: "encryptionkey@0"
I20260926 07:01:05.323398 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1"
format_stamp: "Formatted at 2026-09-26 07:01:05 on dist-test-slave-sfvt"
server_key: "6e1fad34f292b5349032714908476079"
server_key_iv: "bd043f419c8a3023363c1a5de5155855"
server_key_version: "encryptionkey@0"
I20260926 07:01:05.324692 20212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:05.325692 20226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.325874 20212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:05.325965 20212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1"
format_stamp: "Formatted at 2026-09-26 07:01:05 on dist-test-slave-sfvt"
server_key: "6e1fad34f292b5349032714908476079"
server_key_iv: "bd043f419c8a3023363c1a5de5155855"
server_key_version: "encryptionkey@0"
I20260926 07:01:05.326040 20212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:05.336710 20212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:05.337002 20212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:05.337145 20212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:05.337363 20212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:05.337700 20212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:05.337754 20212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.337796 20212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:05.337821 20212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:05.343781 20212 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:39303
I20260926 07:01:05.343837 20339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:39303 every 8 connection(s)
I20260926 07:01:05.344148 20212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 07:01:05.349076 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.344850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:58483 (local address 127.1.74.126:42779)
0926 07:01:05.344926 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.344928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.345172 (+ 244us) server_negotiation.cc:407] Connection header received
0926 07:01:05.345659 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.345660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.345670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.345725 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:05.346359 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.346849 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.347842 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.348033 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.348097 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.348837 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:05.348839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:05.348839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:05.348843 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:05.348856 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:05.348859 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:05.348905 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.348988 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.349013 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:01:05.349251 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.344878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:42779 (local address 127.1.74.67:58483)
0926 07:01:05.345046 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:05.345079 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:05.345575 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:05.345777 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:05.345781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:05.345789 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:05.346280 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.346284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.346931 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.346932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.347744 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.347746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.348113 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.348114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.348221 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.348713 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:01:05.348713 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:05.348740 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:01:05.348763 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:05.348923 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:05.348925 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:05.348927 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:05.348977 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:01:05.349052 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260926 07:01:05.349745 20340 heartbeater.cc:347] Connected to a master server at 127.1.74.126:42779
I20260926 07:01:05.349890 20340 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:05.350098 20340 heartbeater.cc:511] Master 127.1.74.126:42779 requested a full tablet report, sending...
I20260926 07:01:05.350461 19890 ts_manager.cc:194] Registered new tserver with Master: d82e3a0a3cdb443aa6cfd2425a6171a1 (127.1.74.67:39303)
I20260926 07:01:05.350843 19890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.67:58483
I20260926 07:01:05.353600 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 20212
I20260926 07:01:05.353724 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:05.353915 1321 external_mini_cluster.cc:1428] Setting key 4435871ed8b89f1eba185b63226d4a53
I20260926 07:01:05.354868 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:01:05.359216 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.356196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.1.74.126:42779)
0926 07:01:05.356235 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.356238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.356353 (+ 115us) server_negotiation.cc:407] Connection header received
0926 07:01:05.356428 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.356429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.356441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.356490 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:05.356664 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.357131 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.357921 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.358114 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.358191 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.358928 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:05.358929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:05.358930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:05.358934 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:05.358948 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:05.358952 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:05.359060 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.359123 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.359133 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 07:01:05.360944 19890 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:05.362516 19890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:05.368741 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.364571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:39303 (local address 127.0.0.1:59470)
0926 07:01:05.364699 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:05.364738 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:05.364837 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:05.365255 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:05.365258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:05.365260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:05.365323 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.365324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.366692 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.366693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.367770 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.367777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.368446 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.368447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.368525 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.368538 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:01:05.368613 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:01:05.368663 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260926 07:01:05.368752 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.364819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.1.74.67:39303)
0926 07:01:05.365092 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.365103 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.365112 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:05.365142 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.365144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.365155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.365235 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:05.365467 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.366603 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.367861 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.368346 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.368441 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.368535 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.368627 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.368663 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
I20260926 07:01:05.368939 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.364581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:35381 (local address 127.0.0.1:43040)
0926 07:01:05.365026 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:05.365037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:05.365099 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:05.365351 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:05.365352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:05.365354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:05.365410 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.365412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.366365 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.366366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.367651 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.367653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.368652 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.368653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.368713 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.368717 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:05.368767 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:01:05.368833 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":79,"threads_started":1}
I20260926 07:01:05.369158 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.364862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.1.74.65:35381)
0926 07:01:05.365040 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.365045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.365084 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:01:05.365164 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.365167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.365183 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.365277 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:05.365491 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.366276 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.367746 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.368460 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.368535 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.368620 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.369042 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.369083 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260926 07:01:05.369719 20274 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a (DEFAULT_TABLE table=test-table [id=46408289dc1349c78db8a66446b46b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:05.369836 20012 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a (DEFAULT_TABLE table=test-table [id=46408289dc1349c78db8a66446b46b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:05.370054 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.364573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:44125 (local address 127.0.0.1:32980)
0926 07:01:05.364949 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:05.364958 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:05.364995 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:05.365362 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:05.365363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:05.365365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:05.365414 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.365416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.367331 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.367331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.368462 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.368465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.369825 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.369826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.369894 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.369897 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:05.369949 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:01:05.369987 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260926 07:01:05.370097 20274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:05.370169 20012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:05.370363 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.364856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.1.74.66:44125)
0926 07:01:05.365219 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.365223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.365232 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:05.365268 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.365270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.365284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.365352 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:05.365496 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.366609 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.368560 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.369641 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.369709 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.369788 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.370222 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.370264 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":77,"threads_started":1}
I20260926 07:01:05.370958 20143 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a (DEFAULT_TABLE table=test-table [id=46408289dc1349c78db8a66446b46b99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:05.371310 20143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:05.373123 20354 tablet_bootstrap.cc:492] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0: Bootstrap starting.
I20260926 07:01:05.373374 20355 tablet_bootstrap.cc:492] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: Bootstrap starting.
I20260926 07:01:05.373920 20354 tablet_bootstrap.cc:654] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:05.373957 20355 tablet_bootstrap.cc:654] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:05.374266 20355 log.cc:824] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:05.374266 20354 log.cc:824] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:05.374439 20356 tablet_bootstrap.cc:492] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: Bootstrap starting.
I20260926 07:01:05.375056 20354 tablet_bootstrap.cc:492] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0: No bootstrap required, opened a new log
I20260926 07:01:05.375065 20356 tablet_bootstrap.cc:654] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:05.375128 20354 ts_tablet_manager.cc:1397] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:05.375309 20355 tablet_bootstrap.cc:492] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: No bootstrap required, opened a new log
I20260926 07:01:05.375373 20355 ts_tablet_manager.cc:1397] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:05.375448 20356 log.cc:824] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:05.376168 20356 tablet_bootstrap.cc:492] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: No bootstrap required, opened a new log
I20260926 07:01:05.376231 20356 ts_tablet_manager.cc:1397] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 07:01:05.377983 20354 raft_consensus.cc:348] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } }
I20260926 07:01:05.377949 20355 raft_consensus.cc:348] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } }
I20260926 07:01:05.378002 20356 raft_consensus.cc:348] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } }
I20260926 07:01:05.378119 20354 raft_consensus.cc:374] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:05.378119 20355 raft_consensus.cc:374] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:05.378144 20356 raft_consensus.cc:374] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:05.378156 20354 raft_consensus.cc:729] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca4e9b31187457496f7941388db35b0, State: Initialized, Role: FOLLOWER
I20260926 07:01:05.378156 20355 raft_consensus.cc:729] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82e3a0a3cdb443aa6cfd2425a6171a1, State: Initialized, Role: FOLLOWER
I20260926 07:01:05.378185 20356 raft_consensus.cc:729] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969c4eba644e4e1fb47d76103251aac5, State: Initialized, Role: FOLLOWER
I20260926 07:01:05.378283 20355 consensus_queue.cc:260] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } }
I20260926 07:01:05.378299 20354 consensus_queue.cc:260] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } }
I20260926 07:01:05.378362 20356 consensus_queue.cc:260] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } }
I20260926 07:01:05.378543 20354 ts_tablet_manager.cc:1428] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:05.378543 20355 ts_tablet_manager.cc:1428] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:05.378664 20078 heartbeater.cc:503] Master 127.1.74.126:42779 was elected leader, sending a full tablet report...
I20260926 07:01:05.378674 20340 heartbeater.cc:503] Master 127.1.74.126:42779 was elected leader, sending a full tablet report...
I20260926 07:01:05.378698 20209 heartbeater.cc:503] Master 127.1.74.126:42779 was elected leader, sending a full tablet report...
I20260926 07:01:05.378883 20356 ts_tablet_manager.cc:1428] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:01:05.465770 20361 raft_consensus.cc:482] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:05.465939 20361 raft_consensus.cc:504] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } }
I20260926 07:01:05.466351 20361 leader_election.cc:288] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d82e3a0a3cdb443aa6cfd2425a6171a1 (127.1.74.67:39303), 969c4eba644e4e1fb47d76103251aac5 (127.1.74.66:44125)
I20260926 07:01:05.470260 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.466598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:41953 (local address 127.1.74.67:39303)
0926 07:01:05.466656 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.466658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.466825 (+ 167us) server_negotiation.cc:407] Connection header received
0926 07:01:05.466877 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.466878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.466887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.466952 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:05.467106 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.467870 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.469350 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.469895 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.469974 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.469983 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.470146 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.470185 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:01:05.470254 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.466638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:35277 (local address 127.1.74.66:44125)
0926 07:01:05.466685 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.466687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.466697 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:05.466757 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.466758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.466765 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.466877 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:05.467104 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.467875 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.469351 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.469911 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.469974 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.469983 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.470148 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.470189 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:01:05.470281 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.466533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:44125 (local address 127.1.74.65:35277)
0926 07:01:05.466595 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:05.466608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:05.466657 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:05.466891 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:05.466892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:05.466893 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:05.466992 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.466994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.467953 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.467954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.469258 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.469261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.469986 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.469987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.470064 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.470067 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:05.470125 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:01:05.470172 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":57}
I20260926 07:01:05.470296 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.466539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:39303 (local address 127.1.74.65:41953)
0926 07:01:05.466728 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:05.466745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:05.466789 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:05.466961 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:05.466962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:05.466964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:05.467031 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.467033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.467953 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.467954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.469270 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:05.469271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.469986 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.469987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:05.470064 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.470068 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:05.470125 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 07:01:05.470172 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":124,"thread_start_us":82,"threads_started":1}
W20260926 07:01:05.470718 20210 tablet.cc:2369] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:01:05.470675 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0ff70d47874ad4b2f3a28e5b8bd98a" candidate_uuid: "aca4e9b31187457496f7941388db35b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969c4eba644e4e1fb47d76103251aac5" is_pre_election: true
I20260926 07:01:05.470738 20294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0ff70d47874ad4b2f3a28e5b8bd98a" candidate_uuid: "aca4e9b31187457496f7941388db35b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" is_pre_election: true
I20260926 07:01:05.470849 20163 raft_consensus.cc:2454] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca4e9b31187457496f7941388db35b0 in term 0.
I20260926 07:01:05.470903 20294 raft_consensus.cc:2454] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca4e9b31187457496f7941388db35b0 in term 0.
I20260926 07:01:05.471148 19967 leader_election.cc:302] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aca4e9b31187457496f7941388db35b0, d82e3a0a3cdb443aa6cfd2425a6171a1; no voters:
I20260926 07:01:05.471324 20361 raft_consensus.cc:2781] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:05.471401 20361 raft_consensus.cc:482] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:05.471438 20361 raft_consensus.cc:3037] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:05.472146 20361 raft_consensus.cc:504] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } }
I20260926 07:01:05.472306 20361 leader_election.cc:288] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [CANDIDATE]: Term 1 election: Requested vote from peers d82e3a0a3cdb443aa6cfd2425a6171a1 (127.1.74.67:39303), 969c4eba644e4e1fb47d76103251aac5 (127.1.74.66:44125)
I20260926 07:01:05.472450 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0ff70d47874ad4b2f3a28e5b8bd98a" candidate_uuid: "aca4e9b31187457496f7941388db35b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969c4eba644e4e1fb47d76103251aac5"
I20260926 07:01:05.472471 20294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f0ff70d47874ad4b2f3a28e5b8bd98a" candidate_uuid: "aca4e9b31187457496f7941388db35b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1"
I20260926 07:01:05.472538 20294 raft_consensus.cc:3037] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:05.472538 20163 raft_consensus.cc:3037] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:05.473407 20163 raft_consensus.cc:2454] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca4e9b31187457496f7941388db35b0 in term 1.
I20260926 07:01:05.473407 20294 raft_consensus.cc:2454] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca4e9b31187457496f7941388db35b0 in term 1.
I20260926 07:01:05.473565 19967 leader_election.cc:302] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aca4e9b31187457496f7941388db35b0, d82e3a0a3cdb443aa6cfd2425a6171a1; no voters:
I20260926 07:01:05.473656 20361 raft_consensus.cc:2781] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:05.473840 20361 raft_consensus.cc:686] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [term 1 LEADER]: Becoming Leader. State: Replica: aca4e9b31187457496f7941388db35b0, State: Running, Role: LEADER
I20260926 07:01:05.473935 20361 consensus_queue.cc:237] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } }
I20260926 07:01:05.474766 19890 catalog_manager.cc:5495] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to aca4e9b31187457496f7941388db35b0 (127.1.74.65). New cstate: current_term: 1 leader_uuid: "aca4e9b31187457496f7941388db35b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aca4e9b31187457496f7941388db35b0" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 35381 } health_report { overall_health: HEALTHY } } }
I20260926 07:01:05.483186 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.481146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.1.74.65:35381)
0926 07:01:05.481186 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:05.481188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:05.481195 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:01:05.481248 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:05.481249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:05.481265 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:05.481320 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:05.481400 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.481832 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.482502 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:05.482675 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:05.482730 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:05.482863 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:05.482943 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:05.483012 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:05.483052 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 07:01:05.483060 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:01:05.484947 20163 raft_consensus.cc:1261] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [term 1 FOLLOWER]: Refusing update from remote peer aca4e9b31187457496f7941388db35b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:05.484967 20294 raft_consensus.cc:1261] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [term 1 FOLLOWER]: Refusing update from remote peer aca4e9b31187457496f7941388db35b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:05.485313 20364 consensus_queue.cc:1035] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "969c4eba644e4e1fb47d76103251aac5" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 44125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:05.485435 20361 consensus_queue.cc:1035] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P aca4e9b31187457496f7941388db35b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d82e3a0a3cdb443aa6cfd2425a6171a1" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 39303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:05.487614 20367 mvcc.cc:205] Tried to move back new op lower bound from 7333503244224655360 to 7333503244182695936. Current Snapshot: MvccSnapshot[applied={T|T < 7333503244224655360}]
I20260926 07:01:05.488025 20368 mvcc.cc:205] Tried to move back new op lower bound from 7333503244224655360 to 7333503244182695936. Current Snapshot: MvccSnapshot[applied={T|T < 7333503244224655360}]
I20260926 07:01:05.488467 20369 mvcc.cc:205] Tried to move back new op lower bound from 7333503244224655360 to 7333503244182695936. Current Snapshot: MvccSnapshot[applied={T|T < 7333503244224655360}]
I20260926 07:01:05.489993 19890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:05.490082 19890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:05.490846 19890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2736944fe7ad4b248eee9dfd92ffa42e: Sending DeleteTablet for 3 replicas of tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a
I20260926 07:01:05.491189 20143 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:05 UTC) from {username='slave'} at 127.0.0.1:32980
I20260926 07:01:05.491199 20012 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:05 UTC) from {username='slave'} at 127.0.0.1:43040
I20260926 07:01:05.491359 20274 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:05 UTC) from {username='slave'} at 127.0.0.1:59470
I20260926 07:01:05.491544 20377 tablet_replica.cc:330] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: stopping tablet replica
I20260926 07:01:05.491634 20377 raft_consensus.cc:2229] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:05.491633 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 19950
I20260926 07:01:05.492377 20377 raft_consensus.cc:2258] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:05.492391 20379 tablet_replica.cc:330] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: stopping tablet replica
I20260926 07:01:05.492475 20379 raft_consensus.cc:2229] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:05.492563 20379 raft_consensus.cc:2258] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:05.492857 20377 ts_tablet_manager.cc:1905] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:05.492969 20379 ts_tablet_manager.cc:1905] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:05.494731 20379 ts_tablet_manager.cc:1918] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:05.494747 20377 ts_tablet_manager.cc:1918] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:05.494808 20377 log.cc:1196] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/0f0ff70d47874ad4b2f3a28e5b8bd98a
I20260926 07:01:05.494808 20379 log.cc:1196] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/0f0ff70d47874ad4b2f3a28e5b8bd98a
I20260926 07:01:05.495072 20379 ts_tablet_manager.cc:1939] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P d82e3a0a3cdb443aa6cfd2425a6171a1: Deleting consensus metadata
I20260926 07:01:05.495072 20377 ts_tablet_manager.cc:1939] T 0f0ff70d47874ad4b2f3a28e5b8bd98a P 969c4eba644e4e1fb47d76103251aac5: Deleting consensus metadata
I20260926 07:01:05.495455 19876 catalog_manager.cc:4841] TS d82e3a0a3cdb443aa6cfd2425a6171a1 (127.1.74.67:39303): tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a (table test-table [id=46408289dc1349c78db8a66446b46b99]) successfully deleted
I20260926 07:01:05.495489 19875 catalog_manager.cc:4841] TS 969c4eba644e4e1fb47d76103251aac5 (127.1.74.66:44125): tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a (table test-table [id=46408289dc1349c78db8a66446b46b99]) successfully deleted
W20260926 07:01:05.497942 19877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:35381 (error 108)
I20260926 07:01:05.498176 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 20081
I20260926 07:01:05.498768 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:05.498244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:35381 (local address 127.0.0.1:43054)
0926 07:01:05.498682 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:05.498729 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:35381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
W20260926 07:01:05.498931 19877 catalog_manager.cc:4584] TS aca4e9b31187457496f7941388db35b0 (127.1.74.65:35381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f0ff70d47874ad4b2f3a28e5b8bd98a: Network error: Client connection negotiation failed: client connection to 127.1.74.65:35381: connect: Connection refused (error 111)
I20260926 07:01:05.502717 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 20212
I20260926 07:01:05.507458 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 19859
2026-09-26T07:01:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (721 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:05 dist-test-slave-sfvt krb5kdc[20383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:05 dist-test-slave-sfvt krb5kdc[20383](info): set up 2 sockets
Sep 26 07:01:05 dist-test-slave-sfvt krb5kdc[20383](info): commencing operation
krb5kdc: starting...
W20260926 07:01:06.547195 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:06 dist-test-slave-sfvt krb5kdc[20383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:06.672120 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45179
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:45179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:06.751243 20396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:06.751468 20396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:06.751489 20396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:06.752545 20396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:06.752586 20396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:06.752601 20396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:06.752614 20396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:06.752625 20396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:06.754137 20396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:45179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:06.754323 20396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:06.754485 20396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:06.759254 20402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:06.759311 20404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:06.759311 20401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:06.759352 20396 server_base.cc:1034] running on GCE node
I20260926 07:01:06.759718 20396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:06.760027 20396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:06.761202 20396 hybrid_clock.cc:648] HybridClock initialized: now 1790406066761188 us; error 26 us; skew 500 ppm
Sep 26 07:01:06 dist-test-slave-sfvt krb5kdc[20383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406066, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:06.763831 20396 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:06.764238 20396 webserver.cc:460] Webserver started at http://127.1.74.126:41471/ using document root <none> and password file <none>
I20260926 07:01:06.764469 20396 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:06.764535 20396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:06.764686 20396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:06.765724 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "d1d32ad10e184e6f846db73739caf3a3"
format_stamp: "Formatted at 2026-09-26 07:01:06 on dist-test-slave-sfvt"
server_key: "53d269ea077d067e7bec825a1c63b46b"
server_key_iv: "60bf41b6784ecae67c6f070bc930900e"
server_key_version: "encryptionkey@0"
I20260926 07:01:06.766081 20396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "d1d32ad10e184e6f846db73739caf3a3"
format_stamp: "Formatted at 2026-09-26 07:01:06 on dist-test-slave-sfvt"
server_key: "53d269ea077d067e7bec825a1c63b46b"
server_key_iv: "60bf41b6784ecae67c6f070bc930900e"
server_key_version: "encryptionkey@0"
I20260926 07:01:06.767409 20396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:01:06.768105 20411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:06.768280 20396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:06.768375 20396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "d1d32ad10e184e6f846db73739caf3a3"
format_stamp: "Formatted at 2026-09-26 07:01:06 on dist-test-slave-sfvt"
server_key: "53d269ea077d067e7bec825a1c63b46b"
server_key_iv: "60bf41b6784ecae67c6f070bc930900e"
server_key_version: "encryptionkey@0"
I20260926 07:01:06.768461 20396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:06.787699 20396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:06.790071 20396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:06.790239 20396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:06.794510 20396 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:45179
I20260926 07:01:06.794565 20463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:45179 every 8 connection(s)
I20260926 07:01:06.794978 20396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:06.795908 20464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:06.796432 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 20396
I20260926 07:01:06.796550 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:06.796741 1321 external_mini_cluster.cc:1428] Setting key 79f843c02d572c5451c6a87036499e41
I20260926 07:01:06.798964 20464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3: Bootstrap starting.
I20260926 07:01:06.799698 20464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:06.800047 20464 log.cc:824] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:06.800825 20464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3: No bootstrap required, opened a new log
Sep 26 07:01:06 dist-test-slave-sfvt krb5kdc[20383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:06.803503 20464 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d32ad10e184e6f846db73739caf3a3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45179 } }
I20260926 07:01:06.803668 20464 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:06.803714 20464 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d32ad10e184e6f846db73739caf3a3, State: Initialized, Role: FOLLOWER
I20260926 07:01:06.803840 20464 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d32ad10e184e6f846db73739caf3a3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45179 } }
I20260926 07:01:06.803915 20464 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:06.803937 20464 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:06.804005 20464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:06.804778 20464 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d32ad10e184e6f846db73739caf3a3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45179 } }
I20260926 07:01:06.804900 20464 leader_election.cc:302] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d32ad10e184e6f846db73739caf3a3; no voters:
I20260926 07:01:06.805080 20464 leader_election.cc:288] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:06.805130 20469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:06.805267 20469 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [term 1 LEADER]: Becoming Leader. State: Replica: d1d32ad10e184e6f846db73739caf3a3, State: Running, Role: LEADER
I20260926 07:01:06.805393 20464 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:06.805356 20469 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d32ad10e184e6f846db73739caf3a3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45179 } }
I20260926 07:01:06.805754 20470 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d32ad10e184e6f846db73739caf3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d32ad10e184e6f846db73739caf3a3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45179 } } }
I20260926 07:01:06.805755 20471 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d32ad10e184e6f846db73739caf3a3. Latest consensus state: current_term: 1 leader_uuid: "d1d32ad10e184e6f846db73739caf3a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d32ad10e184e6f846db73739caf3a3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45179 } } }
I20260926 07:01:06.805961 20471 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:06.805935 20470 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:06.806353 20478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:06.806919 20478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:06.808717 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:06.797271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.1.74.126:45179)
0926 07:01:06.797437 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:06.797444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:06.797469 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:01:06.797872 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:06.797878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:06.797924 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:06.798041 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:06.798851 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:06.799830 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:06.800829 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:06.801273 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:06.801355 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:06.804194 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:06.804198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:06.804198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:06.804210 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:06.806693 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:06.807233 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:06.807236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:06.807237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:06.807365 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:06.807590 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:06.807591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:06.807591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:06.807892 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:06.808413 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:06.808470 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:01:06.808571 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260926 07:01:06.809186 20478 catalog_manager.cc:1347] Generated new cluster ID: c05f3bdbc78f4aee802f23c7d960ec4f
I20260926 07:01:06.809239 20478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:06.826978 20478 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:06.827749 20478 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:06.829826 20478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3: Generated new TSK 0
I20260926 07:01:06.830029 20478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.74.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
Entry for principal kudu/127.1.74.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab.
I20260926 07:01:06.862226 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45179
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:06.946602 20490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:06.946822 20490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:06.946847 20490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:06.948073 20490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:06.948117 20490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:06.948144 20490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 07:01:06.949620 20490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.65.keytab
--principal=kudu/127.1.74.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:06.949867 20490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:06.950104 20490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:06.955190 20496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:06.955183 20495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:06.955269 20490 server_base.cc:1034] running on GCE node
W20260926 07:01:06.955183 20498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:06.955623 20490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:06.955868 20490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:06.957006 20490 hybrid_clock.cc:648] HybridClock initialized: now 1790406066956989 us; error 40 us; skew 500 ppm
Sep 26 07:01:06 dist-test-slave-sfvt krb5kdc[20383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406066, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:06.959405 20490 init.cc:377] Logged in from keytab as kudu/127.1.74.65@KRBTEST.COM (short username kudu)
I20260926 07:01:06.959837 20490 webserver.cc:460] Webserver started at http://127.1.74.65:33427/ using document root <none> and password file <none>
I20260926 07:01:06.960069 20490 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:06.960139 20490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:06.960294 20490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:06.961365 20490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed"
format_stamp: "Formatted at 2026-09-26 07:01:06 on dist-test-slave-sfvt"
server_key: "d32cee3e6880d2b13ea5067a10084056"
server_key_iv: "6eec55fc4e2c0ee3c2d17ee49207206f"
server_key_version: "encryptionkey@0"
I20260926 07:01:06.961741 20490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed"
format_stamp: "Formatted at 2026-09-26 07:01:06 on dist-test-slave-sfvt"
server_key: "d32cee3e6880d2b13ea5067a10084056"
server_key_iv: "6eec55fc4e2c0ee3c2d17ee49207206f"
server_key_version: "encryptionkey@0"
I20260926 07:01:06.963043 20490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:06.963814 20505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:06.963973 20490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:06.964067 20490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed"
format_stamp: "Formatted at 2026-09-26 07:01:06 on dist-test-slave-sfvt"
server_key: "d32cee3e6880d2b13ea5067a10084056"
server_key_iv: "6eec55fc4e2c0ee3c2d17ee49207206f"
server_key_version: "encryptionkey@0"
I20260926 07:01:06.964151 20490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:06.979234 20490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:06.981137 20490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:06.981302 20490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:06.981541 20490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:06.981837 20490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:06.981889 20490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:06.981933 20490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:06.981958 20490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:06.987898 20490 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:46793
I20260926 07:01:06.987951 20618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:46793 every 8 connection(s)
I20260926 07:01:06.988245 20490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:01:06 dist-test-slave-sfvt krb5kdc[20383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406066, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.65@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:06.997061 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 20490
I20260926 07:01:06.997175 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:06.997381 1321 external_mini_cluster.cc:1428] Setting key f906c41442aaf89b148f2c503a226a7c
I20260926 07:01:06.997593 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:06.988905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:49911 (local address 127.1.74.126:45179)
0926 07:01:06.988947 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:06.988949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:06.989160 (+ 211us) server_negotiation.cc:407] Connection header received
0926 07:01:06.989675 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:06.989676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:06.989684 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:06.989736 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:06.990638 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:06.991123 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:06.992151 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:06.992348 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:06.992398 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:06.994441 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:06.994443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:06.994443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:06.994449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:06.996150 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:06.996507 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:06.996508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:06.996508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:06.996571 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:06.996842 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:06.996844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:06.996844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:06.996966 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:06.997034 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:06.997369 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 07:01:06.997473 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:06.997746 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:06.988873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45179 (local address 127.1.74.65:49911)
0926 07:01:06.989045 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:06.989087 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:06.989555 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:06.989797 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:06.989800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:06.990370 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:06.990565 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:06.990569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:06.991211 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:06.991211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:06.992072 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:06.992075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:06.992418 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:06.992419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:06.992518 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:06.993040 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:06.993049 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:06.994349 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:06.996226 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:06.996232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:06.996233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:06.996439 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:06.996630 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:06.996631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:06.996631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:06.996729 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:06.997048 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:06.997055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:06.997227 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 07:01:06.997352 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 07:01:06.997519 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260926 07:01:06.998239 20619 heartbeater.cc:347] Connected to a master server at 127.1.74.126:45179
I20260926 07:01:06.998346 20619 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:06.998518 20619 heartbeater.cc:511] Master 127.1.74.126:45179 requested a full tablet report, sending...
I20260926 07:01:06.999075 20428 ts_manager.cc:194] Registered new tserver with Master: 40485af5ef8b4c2dbd7e7e5f9d3a58ed (127.1.74.65:46793)
I20260926 07:01:06.999972 20428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.65@KRBTEST.COM'} at 127.1.74.65:49911
WARNING: no policy specified for kudu/127.1.74.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
Entry for principal kudu/127.1.74.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab.
I20260926 07:01:07.022609 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45179
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:07.104732 20624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:07.104976 20624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:07.105023 20624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:07.106349 20624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:07.106410 20624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:07.106458 20624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 07:01:07.108053 20624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.66.keytab
--principal=kudu/127.1.74.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:07.108325 20624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:07.108558 20624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:07.113556 20630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:07.113561 20629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:07.113561 20632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:07.113807 20624 server_base.cc:1034] running on GCE node
I20260926 07:01:07.114025 20624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:07.114267 20624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:07.115427 20624 hybrid_clock.cc:648] HybridClock initialized: now 1790406067115417 us; error 27 us; skew 500 ppm
Sep 26 07:01:07 dist-test-slave-sfvt krb5kdc[20383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406067, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:07.117955 20624 init.cc:377] Logged in from keytab as kudu/127.1.74.66@KRBTEST.COM (short username kudu)
I20260926 07:01:07.118386 20624 webserver.cc:460] Webserver started at http://127.1.74.66:34367/ using document root <none> and password file <none>
I20260926 07:01:07.118614 20624 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:07.118681 20624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:07.118829 20624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:07.119963 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "9ff55a95d2ba4a368d0199e728f73a6d"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "3c29ba122b40700e884960998076d1bf"
server_key_iv: "cf69b25d2d0aa8ffc218b66aa140b6ac"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.120316 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "9ff55a95d2ba4a368d0199e728f73a6d"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "3c29ba122b40700e884960998076d1bf"
server_key_iv: "cf69b25d2d0aa8ffc218b66aa140b6ac"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.121611 20624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:07.122308 20639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.122489 20624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:07.122586 20624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "9ff55a95d2ba4a368d0199e728f73a6d"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "3c29ba122b40700e884960998076d1bf"
server_key_iv: "cf69b25d2d0aa8ffc218b66aa140b6ac"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.122665 20624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:07.136175 20624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:07.138571 20624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:07.138733 20624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:07.138965 20624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:07.139331 20624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:07.139390 20624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.139430 20624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:07.139460 20624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.145119 20624 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:43773
I20260926 07:01:07.145166 20752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:43773 every 8 connection(s)
I20260926 07:01:07.145474 20624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 07:01:07.147907 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 20624
I20260926 07:01:07.148015 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:07.148226 1321 external_mini_cluster.cc:1428] Setting key 16039038016a5a24a2634ab3aa5cfb95
Sep 26 07:01:07 dist-test-slave-sfvt krb5kdc[20383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406067, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.66@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:07.155416 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.146141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:42717 (local address 127.1.74.126:45179)
0926 07:01:07.146208 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.146210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.146395 (+ 185us) server_negotiation.cc:407] Connection header received
0926 07:01:07.146928 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.146930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.146943 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.147001 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:07.148090 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.148741 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.149764 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.149974 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.150054 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.152332 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:07.152334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:07.152334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:07.152343 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:07.153756 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:07.154212 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:07.154213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:07.154214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:07.154286 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:07.154569 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:07.154570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:07.154570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:07.154759 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:07.154838 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.155214 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.155349 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:01:07.155714 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.146117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45179 (local address 127.1.74.66:42717)
0926 07:01:07.146272 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:07.146308 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:07.146815 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:07.147077 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:07.147080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:07.147675 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:07.147995 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.148003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.148846 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.148847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.149660 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.149662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.150058 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.150059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.150190 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.150741 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:07.150750 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:07.152231 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:07.153859 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:07.153867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:07.153869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:07.154100 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:07.154382 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:07.154383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:07.154383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:07.154491 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:07.154873 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:07.154881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:07.155063 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 07:01:07.155188 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 07:01:07.155389 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260926 07:01:07.156286 20753 heartbeater.cc:347] Connected to a master server at 127.1.74.126:45179
I20260926 07:01:07.156383 20753 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:07.156591 20753 heartbeater.cc:511] Master 127.1.74.126:45179 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.74.67@KRBTEST.COM; defaulting to no policy
I20260926 07:01:07.157016 20428 ts_manager.cc:194] Registered new tserver with Master: 9ff55a95d2ba4a368d0199e728f73a6d (127.1.74.66:43773)
I20260926 07:01:07.157403 20428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.66@KRBTEST.COM'} at 127.1.74.66:42717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
Entry for principal kudu/127.1.74.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab.
I20260926 07:01:07.181826 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45179
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:07.263010 20758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:07.263212 20758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:07.263234 20758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:07.264395 20758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:07.264438 20758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:07.264493 20758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 07:01:07.265930 20758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.67.keytab
--principal=kudu/127.1.74.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:45179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:07.266161 20758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:07.266408 20758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:07.271325 20764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:07.271430 20763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:07.271485 20758 server_base.cc:1034] running on GCE node
W20260926 07:01:07.271328 20766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:07.271883 20758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:07.272140 20758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:07.273298 20758 hybrid_clock.cc:648] HybridClock initialized: now 1790406067273276 us; error 31 us; skew 500 ppm
Sep 26 07:01:07 dist-test-slave-sfvt krb5kdc[20383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406067, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:07.275786 20758 init.cc:377] Logged in from keytab as kudu/127.1.74.67@KRBTEST.COM (short username kudu)
I20260926 07:01:07.276258 20758 webserver.cc:460] Webserver started at http://127.1.74.67:35701/ using document root <none> and password file <none>
I20260926 07:01:07.276505 20758 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:07.276573 20758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:07.276728 20758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:07.277787 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "5251b7cf897641778c5c03274f135621"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "6555a8a3b1c8029bbeaa0241941a2969"
server_key_iv: "f7678bbe35ad7c330b4dcb71ba71358c"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.278159 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "5251b7cf897641778c5c03274f135621"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "6555a8a3b1c8029bbeaa0241941a2969"
server_key_iv: "f7678bbe35ad7c330b4dcb71ba71358c"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.279492 20758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:01:07.280220 20773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.280395 20758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:07.280489 20758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "5251b7cf897641778c5c03274f135621"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "6555a8a3b1c8029bbeaa0241941a2969"
server_key_iv: "f7678bbe35ad7c330b4dcb71ba71358c"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.280568 20758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:07.311347 20758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:07.313460 20758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:07.313637 20758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:07.313879 20758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:07.314188 20758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:07.314220 20758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.314265 20758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:07.314291 20758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.320358 20758 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:41723
I20260926 07:01:07.320401 20886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:41723 every 8 connection(s)
I20260926 07:01:07.320730 20758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:01:07 dist-test-slave-sfvt krb5kdc[20383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406067, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.67@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:07.327121 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 20758
I20260926 07:01:07.327221 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:07.327456 1321 external_mini_cluster.cc:1428] Setting key 4f7f82899be228b19480286bbe300343
I20260926 07:01:07.330408 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.321463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:42055 (local address 127.1.74.126:45179)
0926 07:01:07.321530 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.321532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.321736 (+ 204us) server_negotiation.cc:407] Connection header received
0926 07:01:07.322281 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.322282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.322296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.322342 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:07.323365 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.323873 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.324896 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.325126 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.325207 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.327486 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:07.327488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:07.327488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:07.327496 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:07.328987 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:07.329371 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:07.329372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:07.329372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:07.329421 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:07.329631 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:07.329631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:07.329632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:07.329789 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:07.329879 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.330219 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.330311 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:01:07.330569 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.321422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45179 (local address 127.1.74.67:42055)
0926 07:01:07.321613 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:07.321652 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:07.322163 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:07.322433 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:07.322437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:07.323000 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:07.323200 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.323206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.323987 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.323987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.324800 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.324802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.325230 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.325231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.325355 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.325839 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:07.325848 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:07.327375 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:07.329057 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:07.329063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:07.329065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:07.329293 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:07.329476 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:07.329477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:07.329477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:07.329570 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:07.329895 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:07.329902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:07.330064 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 07:01:07.330177 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:01:07.330336 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260926 07:01:07.331030 20887 heartbeater.cc:347] Connected to a master server at 127.1.74.126:45179
I20260926 07:01:07.331132 20887 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:07.331355 20887 heartbeater.cc:511] Master 127.1.74.126:45179 requested a full tablet report, sending...
I20260926 07:01:07.331771 20428 ts_manager.cc:194] Registered new tserver with Master: 5251b7cf897641778c5c03274f135621 (127.1.74.67:41723)
I20260926 07:01:07.332168 20428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.74.67@KRBTEST.COM'} at 127.1.74.67:42055
I20260926 07:01:07.339344 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:01:07.346947 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.340646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.1.74.126:45179)
0926 07:01:07.340731 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.340733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.340830 (+ 97us) server_negotiation.cc:407] Connection header received
0926 07:01:07.340920 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.340921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.340932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.340989 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:07.341563 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.342056 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.342786 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.342996 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.343068 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.344339 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:07.344341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:07.344341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:07.344348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:07.345733 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:07.346117 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:07.346118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:07.346118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:07.346175 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:07.346385 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:07.346386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:07.346386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:07.346538 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:07.346641 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.346754 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.346841 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:01:07.348765 20428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:07.350323 20428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:07.356946 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.352676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:43773 (local address 127.0.0.1:42550)
0926 07:01:07.352923 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:07.352935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:07.352987 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:07.353378 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:07.353379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:07.353381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:07.353436 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.353438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.354399 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.354400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.355758 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.355760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.356704 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.356710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.356781 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.356784 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:07.356833 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 07:01:07.356874 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260926 07:01:07.357038 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.352870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.1.74.67:41723)
0926 07:01:07.353133 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.353137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.353145 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:07.353177 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.353178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.353191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.353282 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:07.353466 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.354564 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.355951 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.356491 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.356591 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.356653 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.356915 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.356951 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
I20260926 07:01:07.356992 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.353057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.1.74.66:43773)
0926 07:01:07.353217 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.353222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.353233 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:01:07.353277 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.353279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.353294 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.353383 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:07.353494 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.354280 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.355845 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.356595 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.356702 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.356777 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.356846 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.356888 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260926 07:01:07.357134 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.352676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:41723 (local address 127.0.0.1:55318)
0926 07:01:07.352844 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:07.352882 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:07.352949 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:07.353291 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:07.353295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:07.353297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:07.353380 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.353382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.354792 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.354793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.355871 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.355873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.356595 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.356596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.356677 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.356692 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 07:01:07.356765 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 07:01:07.356827 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260926 07:01:07.357600 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.352700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:46793 (local address 127.0.0.1:47390)
0926 07:01:07.353224 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:07.353234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:07.353283 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:07.353631 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:07.353632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:07.353635 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:07.353675 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.353676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.354867 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.354868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.355928 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.355930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.356854 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.356855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.356911 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.356916 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:07.356966 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:01:07.357532 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":83,"threads_started":1}
I20260926 07:01:07.357818 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.352933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.1.74.65:46793)
0926 07:01:07.353412 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.353417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.353426 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:07.353469 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.353471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.353486 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.353557 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:07.353749 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.354783 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.356018 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.356715 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.356772 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.356820 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.357702 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.357746 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":82,"threads_started":1}
I20260926 07:01:07.357865 20821 tablet_service.cc:1467] Processing CreateTablet for tablet 8ccee264a81e4e2087d361bd4c49da58 (DEFAULT_TABLE table=test-table [id=903d34ffe072498f9d11ef4dc6aff47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:07.357940 20687 tablet_service.cc:1467] Processing CreateTablet for tablet 8ccee264a81e4e2087d361bd4c49da58 (DEFAULT_TABLE table=test-table [id=903d34ffe072498f9d11ef4dc6aff47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:07.358202 20687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ccee264a81e4e2087d361bd4c49da58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:07.358222 20821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ccee264a81e4e2087d361bd4c49da58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:07.359959 20553 tablet_service.cc:1467] Processing CreateTablet for tablet 8ccee264a81e4e2087d361bd4c49da58 (DEFAULT_TABLE table=test-table [id=903d34ffe072498f9d11ef4dc6aff47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:07.360282 20553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ccee264a81e4e2087d361bd4c49da58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:07.360994 20904 tablet_bootstrap.cc:492] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: Bootstrap starting.
I20260926 07:01:07.361356 20905 tablet_bootstrap.cc:492] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: Bootstrap starting.
I20260926 07:01:07.361686 20904 tablet_bootstrap.cc:654] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:07.362061 20905 tablet_bootstrap.cc:654] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:07.362398 20904 log.cc:824] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:07.362591 20905 log.cc:824] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:07.363201 20904 tablet_bootstrap.cc:492] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: No bootstrap required, opened a new log
I20260926 07:01:07.363317 20904 ts_tablet_manager.cc:1397] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:07.363397 20905 tablet_bootstrap.cc:492] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: No bootstrap required, opened a new log
I20260926 07:01:07.363463 20905 ts_tablet_manager.cc:1397] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:07.363643 20908 tablet_bootstrap.cc:492] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed: Bootstrap starting.
I20260926 07:01:07.364358 20908 tablet_bootstrap.cc:654] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:07.364990 20908 log.cc:824] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:07.365187 20905 raft_consensus.cc:348] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } }
I20260926 07:01:07.365417 20905 raft_consensus.cc:374] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:07.365469 20905 raft_consensus.cc:729] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5251b7cf897641778c5c03274f135621, State: Initialized, Role: FOLLOWER
I20260926 07:01:07.365482 20904 raft_consensus.cc:348] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } }
I20260926 07:01:07.365664 20904 raft_consensus.cc:374] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:07.365703 20904 raft_consensus.cc:729] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff55a95d2ba4a368d0199e728f73a6d, State: Initialized, Role: FOLLOWER
I20260926 07:01:07.365664 20905 consensus_queue.cc:260] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } }
I20260926 07:01:07.365821 20904 consensus_queue.cc:260] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } }
I20260926 07:01:07.365954 20905 ts_tablet_manager.cc:1428] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:01:07.366060 20904 ts_tablet_manager.cc:1428] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:07.366070 20887 heartbeater.cc:503] Master 127.1.74.126:45179 was elected leader, sending a full tablet report...
I20260926 07:01:07.365861 20908 tablet_bootstrap.cc:492] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed: No bootstrap required, opened a new log
I20260926 07:01:07.366179 20753 heartbeater.cc:503] Master 127.1.74.126:45179 was elected leader, sending a full tablet report...
I20260926 07:01:07.366197 20908 ts_tablet_manager.cc:1397] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:01:07.368153 20908 raft_consensus.cc:348] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } }
I20260926 07:01:07.368288 20908 raft_consensus.cc:374] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:07.368310 20908 raft_consensus.cc:729] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40485af5ef8b4c2dbd7e7e5f9d3a58ed, State: Initialized, Role: FOLLOWER
I20260926 07:01:07.368430 20908 consensus_queue.cc:260] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } }
I20260926 07:01:07.368727 20908 ts_tablet_manager.cc:1428] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:01:07.368752 20619 heartbeater.cc:503] Master 127.1.74.126:45179 was elected leader, sending a full tablet report...
W20260926 07:01:07.396026 20754 tablet.cc:2369] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:01:07.488969 20620 tablet.cc:2369] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:01:07.571290 20888 tablet.cc:2369] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:01:07.588685 20911 raft_consensus.cc:482] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:07.588805 20911 raft_consensus.cc:504] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } }
I20260926 07:01:07.589242 20911 leader_election.cc:288] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5251b7cf897641778c5c03274f135621 (127.1.74.67:41723), 40485af5ef8b4c2dbd7e7e5f9d3a58ed (127.1.74.65:46793)
I20260926 07:01:07.592181 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.589525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:42225 (local address 127.1.74.65:46793)
0926 07:01:07.589576 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.589578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.589733 (+ 155us) server_negotiation.cc:407] Connection header received
0926 07:01:07.589777 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.589778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.589792 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.589843 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:07.590019 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.590498 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.591417 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.591792 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.591848 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.591856 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.592053 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.592101 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:01:07.592237 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.589560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:39429 (local address 127.1.74.67:41723)
0926 07:01:07.589605 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.589607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.589616 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:07.589647 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.589647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.589658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.589737 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:07.589898 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.590410 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.591489 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.591878 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.591947 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.591958 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.592124 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.592165 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:07.592195 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.589438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:46793 (local address 127.1.74.66:42225)
0926 07:01:07.589652 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:07.589662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:07.589720 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:07.589876 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:07.589877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:07.589879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:07.589948 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.589950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.590577 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.590578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.591341 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.591343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.591878 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.591879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.591970 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.591975 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:07.592028 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 07:01:07.592058 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":53,"thread_start_us":97,"threads_started":1}
I20260926 07:01:07.592463 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.589433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:41723 (local address 127.1.74.66:39429)
0926 07:01:07.589487 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:07.589501 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:07.589549 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:07.589743 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:07.589744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:07.589746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:07.589821 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.589823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.590484 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.590485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.591386 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.591389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.591959 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.591960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.592039 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.592042 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:07.592097 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 07:01:07.592130 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":21}
I20260926 07:01:07.592628 20573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ccee264a81e4e2087d361bd4c49da58" candidate_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" is_pre_election: true
I20260926 07:01:07.592783 20573 raft_consensus.cc:2454] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff55a95d2ba4a368d0199e728f73a6d in term 0.
I20260926 07:01:07.592844 20841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ccee264a81e4e2087d361bd4c49da58" candidate_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5251b7cf897641778c5c03274f135621" is_pre_election: true
I20260926 07:01:07.593001 20841 raft_consensus.cc:2454] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ff55a95d2ba4a368d0199e728f73a6d in term 0.
I20260926 07:01:07.593019 20643 leader_election.cc:302] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40485af5ef8b4c2dbd7e7e5f9d3a58ed, 9ff55a95d2ba4a368d0199e728f73a6d; no voters:
I20260926 07:01:07.593139 20911 raft_consensus.cc:2781] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:07.593201 20911 raft_consensus.cc:482] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:07.593225 20911 raft_consensus.cc:3037] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:07.593918 20911 raft_consensus.cc:504] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } }
I20260926 07:01:07.594055 20911 leader_election.cc:288] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [CANDIDATE]: Term 1 election: Requested vote from peers 5251b7cf897641778c5c03274f135621 (127.1.74.67:41723), 40485af5ef8b4c2dbd7e7e5f9d3a58ed (127.1.74.65:46793)
I20260926 07:01:07.594223 20573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ccee264a81e4e2087d361bd4c49da58" candidate_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed"
I20260926 07:01:07.594223 20841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ccee264a81e4e2087d361bd4c49da58" candidate_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5251b7cf897641778c5c03274f135621"
I20260926 07:01:07.594288 20573 raft_consensus.cc:3037] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:07.594321 20841 raft_consensus.cc:3037] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:07.594964 20573 raft_consensus.cc:2454] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff55a95d2ba4a368d0199e728f73a6d in term 1.
I20260926 07:01:07.595072 20841 raft_consensus.cc:2454] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ff55a95d2ba4a368d0199e728f73a6d in term 1.
I20260926 07:01:07.595119 20643 leader_election.cc:302] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40485af5ef8b4c2dbd7e7e5f9d3a58ed, 9ff55a95d2ba4a368d0199e728f73a6d; no voters:
I20260926 07:01:07.595224 20911 raft_consensus.cc:2781] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:07.595417 20911 raft_consensus.cc:686] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 1 LEADER]: Becoming Leader. State: Replica: 9ff55a95d2ba4a368d0199e728f73a6d, State: Running, Role: LEADER
I20260926 07:01:07.595516 20911 consensus_queue.cc:237] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } }
I20260926 07:01:07.596328 20428 catalog_manager.cc:5495] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ff55a95d2ba4a368d0199e728f73a6d (127.1.74.66). New cstate: current_term: 1 leader_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ff55a95d2ba4a368d0199e728f73a6d" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 43773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 } health_report { overall_health: UNKNOWN } } }
I20260926 07:01:07.640266 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.637816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.1.74.66:43773)
0926 07:01:07.637889 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.637891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.637901 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:07.637944 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.637945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.637968 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.638045 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:07.638202 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.638679 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.639471 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.639668 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.639743 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.639953 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:07.640077 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:07.640146 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.640183 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.640205 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:01:07.642059 20573 raft_consensus.cc:1261] T 8ccee264a81e4e2087d361bd4c49da58 P 40485af5ef8b4c2dbd7e7e5f9d3a58ed [term 1 FOLLOWER]: Refusing update from remote peer 9ff55a95d2ba4a368d0199e728f73a6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:07.642059 20841 raft_consensus.cc:1261] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [term 1 FOLLOWER]: Refusing update from remote peer 9ff55a95d2ba4a368d0199e728f73a6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:07.642508 20914 consensus_queue.cc:1035] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5251b7cf897641778c5c03274f135621" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:07.642680 20911 consensus_queue.cc:1035] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "40485af5ef8b4c2dbd7e7e5f9d3a58ed" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 46793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:07.645187 20919 mvcc.cc:205] Tried to move back new op lower bound from 7333503253059350528 to 7333503252872753152. Current Snapshot: MvccSnapshot[applied={T|T < 7333503253059350528}]
I20260926 07:01:07.645522 20917 mvcc.cc:205] Tried to move back new op lower bound from 7333503253059350528 to 7333503252872753152. Current Snapshot: MvccSnapshot[applied={T|T < 7333503253059350528 or (T in {7333503253059350528})}]
I20260926 07:01:07.646116 20918 mvcc.cc:205] Tried to move back new op lower bound from 7333503253059350528 to 7333503252872753152. Current Snapshot: MvccSnapshot[applied={T|T < 7333503253059350528}]
I20260926 07:01:07.647212 20428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:07.647338 20428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:07.648100 20428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1d32ad10e184e6f846db73739caf3a3: Sending DeleteTablet for 3 replicas of tablet 8ccee264a81e4e2087d361bd4c49da58
I20260926 07:01:07.648420 20821 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ccee264a81e4e2087d361bd4c49da58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:07 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:55318
I20260926 07:01:07.648597 20553 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ccee264a81e4e2087d361bd4c49da58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:07 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:47390
I20260926 07:01:07.648643 20927 tablet_replica.cc:330] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: stopping tablet replica
I20260926 07:01:07.648624 20687 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ccee264a81e4e2087d361bd4c49da58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:07 UTC) from {username='kudu', principal='kudu/127.1.74.126@KRBTEST.COM'} at 127.0.0.1:42550
I20260926 07:01:07.648752 20927 raft_consensus.cc:2229] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:07.648849 20927 raft_consensus.cc:2258] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:07.648989 20929 tablet_replica.cc:330] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: stopping tablet replica
I20260926 07:01:07.649088 20929 raft_consensus.cc:2229] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:01:07.649181 20929 raft_consensus.cc:2258] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:07.649191 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 20490
I20260926 07:01:07.650470 20927 ts_tablet_manager.cc:1905] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:07.650558 20929 ts_tablet_manager.cc:1905] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:07.652532 20929 ts_tablet_manager.cc:1918] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:07.652532 20927 ts_tablet_manager.cc:1918] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:07.652661 20927 log.cc:1196] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/8ccee264a81e4e2087d361bd4c49da58
I20260926 07:01:07.652673 20929 log.cc:1196] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790405833506994-1321-0/minicluster-data/ts-1/wal/wals/8ccee264a81e4e2087d361bd4c49da58
I20260926 07:01:07.652961 20929 ts_tablet_manager.cc:1939] T 8ccee264a81e4e2087d361bd4c49da58 P 9ff55a95d2ba4a368d0199e728f73a6d: Deleting consensus metadata
I20260926 07:01:07.653086 20927 ts_tablet_manager.cc:1939] T 8ccee264a81e4e2087d361bd4c49da58 P 5251b7cf897641778c5c03274f135621: Deleting consensus metadata
I20260926 07:01:07.653410 20414 catalog_manager.cc:4841] TS 9ff55a95d2ba4a368d0199e728f73a6d (127.1.74.66:43773): tablet 8ccee264a81e4e2087d361bd4c49da58 (table test-table [id=903d34ffe072498f9d11ef4dc6aff47e]) successfully deleted
I20260926 07:01:07.653564 20412 catalog_manager.cc:4841] TS 5251b7cf897641778c5c03274f135621 (127.1.74.67:41723): tablet 8ccee264a81e4e2087d361bd4c49da58 (table test-table [id=903d34ffe072498f9d11ef4dc6aff47e]) successfully deleted
W20260926 07:01:07.656023 20415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:46793 (error 108)
I20260926 07:01:07.656602 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.656457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:46793 (local address 127.0.0.1:47398)
0926 07:01:07.656521 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:07.656561 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:46793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:01:07.656635 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 20624
W20260926 07:01:07.657239 20415 catalog_manager.cc:4584] TS 40485af5ef8b4c2dbd7e7e5f9d3a58ed (127.1.74.65:46793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ccee264a81e4e2087d361bd4c49da58: Network error: Client connection negotiation failed: client connection to 127.1.74.65:46793: connect: Connection refused (error 111)
I20260926 07:01:07.662091 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 20758
I20260926 07:01:07.666832 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 20396
2026-09-26T07:01:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2159 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T07:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:07Z Disabled control of system clock
I20260926 07:01:07.694010 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33857
--webserver_interface=127.1.74.126
--webserver_port=0
--builtin_ntp_servers=127.1.74.84:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:33857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:01:07.774338 20934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:07.774549 20934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:07.774578 20934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:07.775777 20934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:07.775820 20934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:07.775832 20934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:07.775846 20934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:07.775857 20934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:07.777259 20934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:33857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:33857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:07.777504 20934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:07.777837 20934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:07.782861 20942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:07.782850 20939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:07.782958 20940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:07.783000 20934 server_base.cc:1034] running on GCE node
I20260926 07:01:07.783439 20934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:07.783740 20934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:07.784909 20934 hybrid_clock.cc:648] HybridClock initialized: now 1790406067784881 us; error 40 us; skew 500 ppm
I20260926 07:01:07.786337 20934 webserver.cc:460] Webserver started at http://127.1.74.126:34421/ using document root <none> and password file <none>
I20260926 07:01:07.786599 20934 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:07.786648 20934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:07.786826 20934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:07.787989 20934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "0304a83e5bb6446c9a57787b8fe05a3c"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "603d714f1bf19e9dd4da426ed16d6095"
server_key_iv: "525b47af9a5915a0e039680e3dba8ee6"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.788359 20934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "0304a83e5bb6446c9a57787b8fe05a3c"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "603d714f1bf19e9dd4da426ed16d6095"
server_key_iv: "525b47af9a5915a0e039680e3dba8ee6"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.789700 20934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:07.790434 20948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.790666 20934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:07.790757 20934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "0304a83e5bb6446c9a57787b8fe05a3c"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "603d714f1bf19e9dd4da426ed16d6095"
server_key_iv: "525b47af9a5915a0e039680e3dba8ee6"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.790834 20934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:07.800604 20934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:07.800870 20934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:07.801014 20934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:07.805177 20934 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:33857
I20260926 07:01:07.805214 21000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:33857 every 8 connection(s)
I20260926 07:01:07.805589 20934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:07.806581 21001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:07.808255 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 20934
I20260926 07:01:07.808365 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:07.808580 1321 external_mini_cluster.cc:1428] Setting key 4a175b6531dbb4b7fef06844fb474abf
I20260926 07:01:07.809476 21001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c: Bootstrap starting.
I20260926 07:01:07.810165 21001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:07.810511 21001 log.cc:824] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:07.811225 21001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c: No bootstrap required, opened a new log
I20260926 07:01:07.813906 21001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0304a83e5bb6446c9a57787b8fe05a3c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33857 } }
I20260926 07:01:07.814055 21001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:07.814080 21001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0304a83e5bb6446c9a57787b8fe05a3c, State: Initialized, Role: FOLLOWER
I20260926 07:01:07.814249 21001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0304a83e5bb6446c9a57787b8fe05a3c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33857 } }
I20260926 07:01:07.814343 21001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:07.814406 21001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:07.814463 21001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:07.814646 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.809159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.1.74.126:33857)
0926 07:01:07.809343 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.809350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.809374 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:01:07.809833 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.809840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.809906 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.810018 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:07.810310 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.811520 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.812650 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.813036 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.813126 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.813879 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:07.813883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:07.813883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:07.814037 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:07.814100 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:07.814116 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:07.814179 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.814281 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.814365 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260926 07:01:07.815209 21001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0304a83e5bb6446c9a57787b8fe05a3c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33857 } }
I20260926 07:01:07.815341 21001 leader_election.cc:302] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0304a83e5bb6446c9a57787b8fe05a3c; no voters:
I20260926 07:01:07.815539 21001 leader_election.cc:288] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:07.815567 21006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:07.815721 21006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [term 1 LEADER]: Becoming Leader. State: Replica: 0304a83e5bb6446c9a57787b8fe05a3c, State: Running, Role: LEADER
I20260926 07:01:07.815843 21006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0304a83e5bb6446c9a57787b8fe05a3c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33857 } }
I20260926 07:01:07.815886 21001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:07.816279 21007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0304a83e5bb6446c9a57787b8fe05a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0304a83e5bb6446c9a57787b8fe05a3c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33857 } } }
I20260926 07:01:07.816288 21008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0304a83e5bb6446c9a57787b8fe05a3c. Latest consensus state: current_term: 1 leader_uuid: "0304a83e5bb6446c9a57787b8fe05a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0304a83e5bb6446c9a57787b8fe05a3c" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 33857 } } }
I20260926 07:01:07.816514 21007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:07.816548 21008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:07.817083 21015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:07.817585 21015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:07.819063 21015 catalog_manager.cc:1347] Generated new cluster ID: 7ad67c4c71444e4a8614bd98f080a608
I20260926 07:01:07.819131 21015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:07.826062 21015 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:07.826599 21015 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:07.832901 21015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c: Generated new TSK 0
I20260926 07:01:07.833108 21015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:07.838857 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.65:0
--local_ip_for_outbound_sockets=127.1.74.65
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33857
--builtin_ntp_servers=127.1.74.84:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:01:07.920285 21025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:07.920480 21025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:07.920506 21025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:07.921654 21025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:07.921692 21025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:07.921717 21025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.65
I20260926 07:01:07.923043 21025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.74.65
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.65
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:07.923295 21025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:07.923615 21025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:07.928510 21033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:07.928519 21030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:07.928727 21025 server_base.cc:1034] running on GCE node
W20260926 07:01:07.928597 21031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:07.929049 21025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:07.929291 21025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:07.930451 21025 hybrid_clock.cc:648] HybridClock initialized: now 1790406067930441 us; error 29 us; skew 500 ppm
I20260926 07:01:07.931754 21025 webserver.cc:460] Webserver started at http://127.1.74.65:45379/ using document root <none> and password file <none>
I20260926 07:01:07.932024 21025 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:07.932097 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:07.932245 21025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:07.933279 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/data/instance:
uuid: "2730f47b54cf4d26933e1b7cc471a6af"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "0192cc365b5fdeeee76618741947ab71"
server_key_iv: "8b7c326edad3eb9f67ed32b3f6657227"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.933630 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance:
uuid: "2730f47b54cf4d26933e1b7cc471a6af"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "0192cc365b5fdeeee76618741947ab71"
server_key_iv: "8b7c326edad3eb9f67ed32b3f6657227"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.934932 21025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:07.935750 21039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.935911 21025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.936024 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/wal
uuid: "2730f47b54cf4d26933e1b7cc471a6af"
format_stamp: "Formatted at 2026-09-26 07:01:07 on dist-test-slave-sfvt"
server_key: "0192cc365b5fdeeee76618741947ab71"
server_key_iv: "8b7c326edad3eb9f67ed32b3f6657227"
server_key_version: "encryptionkey@0"
I20260926 07:01:07.936116 21025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:07.951116 21025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:07.951467 21025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:07.951627 21025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:07.951866 21025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:07.952214 21025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:07.952266 21025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.952307 21025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:07.952337 21025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:07.958055 21025 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.65:42797
I20260926 07:01:07.958132 21152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.65:42797 every 8 connection(s)
I20260926 07:01:07.958469 21025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/data/info.pb
I20260926 07:01:07.963817 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.959211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.65:50103 (local address 127.1.74.126:33857)
0926 07:01:07.959298 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:07.959300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:07.959528 (+ 228us) server_negotiation.cc:407] Connection header received
0926 07:01:07.960072 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:07.960073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:07.960085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:07.960144 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:07.960841 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.961351 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.962353 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.962577 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.962663 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.963536 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:07.963537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:07.963538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:07.963543 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:07.963557 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:07.963562 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:07.963605 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:07.963707 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:01:07.963736 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:01:07.964049 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:07.959187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33857 (local address 127.1.74.65:50103)
0926 07:01:07.959403 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:07.959437 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:07.959955 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:07.960214 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:07.960218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:07.960228 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:07.960743 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.960748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.961452 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.961453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.962250 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:07.962252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:07.962674 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:07.962675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:07.962809 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:07.963391 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:01:07.963391 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:07.963420 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:01:07.963440 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:07.963616 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:07.963620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:07.963624 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:07.963709 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:01:07.963807 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260926 07:01:07.964026 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 21025
I20260926 07:01:07.964167 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:07.964366 1321 external_mini_cluster.cc:1428] Setting key 2bb8e61c7175f4c4cd4c325e336d815b
I20260926 07:01:07.964541 21153 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33857
I20260926 07:01:07.964635 21153 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:07.964835 21153 heartbeater.cc:511] Master 127.1.74.126:33857 requested a full tablet report, sending...
I20260926 07:01:07.965281 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.66:0
--local_ip_for_outbound_sockets=127.1.74.66
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33857
--builtin_ntp_servers=127.1.74.84:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:01:07.965348 20965 ts_manager.cc:194] Registered new tserver with Master: 2730f47b54cf4d26933e1b7cc471a6af (127.1.74.65:42797)
I20260926 07:01:07.965955 20965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.65:50103
W20260926 07:01:08.046895 21156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:08.047116 21156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:08.047138 21156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:08.048383 21156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:08.048432 21156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:08.048569 21156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.66
I20260926 07:01:08.049994 21156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.74.66
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.66
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:08.050251 21156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:08.050570 21156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:08.055593 21162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:08.055720 21161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:08.055588 21164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:08.055831 21156 server_base.cc:1034] running on GCE node
I20260926 07:01:08.056072 21156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:08.056353 21156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:08.057520 21156 hybrid_clock.cc:648] HybridClock initialized: now 1790406068057502 us; error 30 us; skew 500 ppm
I20260926 07:01:08.058708 21156 webserver.cc:460] Webserver started at http://127.1.74.66:39549/ using document root <none> and password file <none>
I20260926 07:01:08.058959 21156 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:08.059007 21156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:08.059158 21156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:08.060314 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/data/instance:
uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a"
format_stamp: "Formatted at 2026-09-26 07:01:08 on dist-test-slave-sfvt"
server_key: "bf775ac9d40c45ee94a6f906eabed60f"
server_key_iv: "adcb4969b07b1b86da279d3a36814a87"
server_key_version: "encryptionkey@0"
I20260926 07:01:08.060716 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance:
uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a"
format_stamp: "Formatted at 2026-09-26 07:01:08 on dist-test-slave-sfvt"
server_key: "bf775ac9d40c45ee94a6f906eabed60f"
server_key_iv: "adcb4969b07b1b86da279d3a36814a87"
server_key_version: "encryptionkey@0"
I20260926 07:01:08.062012 21156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:01:08.062767 21170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:08.062942 21156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:08.063038 21156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/wal
uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a"
format_stamp: "Formatted at 2026-09-26 07:01:08 on dist-test-slave-sfvt"
server_key: "bf775ac9d40c45ee94a6f906eabed60f"
server_key_iv: "adcb4969b07b1b86da279d3a36814a87"
server_key_version: "encryptionkey@0"
I20260926 07:01:08.063126 21156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:08.091954 21156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:08.092283 21156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:08.092445 21156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:08.092691 21156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:08.093027 21156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:08.093087 21156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:08.093132 21156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:08.093165 21156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:08.098955 21156 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.66:33081
I20260926 07:01:08.099012 21283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.66:33081 every 8 connection(s)
I20260926 07:01:08.099435 21156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/data/info.pb
I20260926 07:01:08.100114 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 21156
I20260926 07:01:08.100227 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:08.100436 1321 external_mini_cluster.cc:1428] Setting key 955d70e3fe266fc4be8cd32cc094fc25
I20260926 07:01:08.101406 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.74.67:0
--local_ip_for_outbound_sockets=127.1.74.67
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33857
--builtin_ntp_servers=127.1.74.84:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:01:08.105448 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.100205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:60717 (local address 127.1.74.126:33857)
0926 07:01:08.100261 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:08.100264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:08.100579 (+ 315us) server_negotiation.cc:407] Connection header received
0926 07:01:08.101282 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:08.101283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:08.101302 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:08.101364 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:08.102307 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.102799 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.103931 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.104169 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.104244 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.105151 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:08.105153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:08.105153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:08.105159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:08.105174 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:08.105178 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:08.105236 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:08.105336 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:01:08.105364 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:08.105684 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.100154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33857 (local address 127.1.74.66:60717)
0926 07:01:08.100432 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:08.100478 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:08.101153 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:08.101450 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:08.101456 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:08.101467 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:08.102190 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.102198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.102927 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.102928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.103822 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.103825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.104266 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.104268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.104405 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.104991 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:01:08.104992 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:08.105022 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:01:08.105045 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:08.105244 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:08.105249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:08.105252 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:08.105342 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:01:08.105449 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
I20260926 07:01:08.106220 21284 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33857
I20260926 07:01:08.106302 21284 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:08.106509 21284 heartbeater.cc:511] Master 127.1.74.126:33857 requested a full tablet report, sending...
I20260926 07:01:08.106921 20965 ts_manager.cc:194] Registered new tserver with Master: 7ef89e9c5ce942d6b09d50d14eb0d98a (127.1.74.66:33081)
I20260926 07:01:08.107368 20965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.66:60717
W20260926 07:01:08.184209 21287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:08.184443 21287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:08.184476 21287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:08.185638 21287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:08.185701 21287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:08.185740 21287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.74.67
I20260926 07:01:08.187060 21287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.74.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.74.67
--webserver_port=0
--tserver_master_addrs=127.1.74.126:33857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.74.67
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:08.187345 21287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:08.187649 21287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:08.193066 21295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:08.193086 21293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:08.193074 21292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:08.193347 21287 server_base.cc:1034] running on GCE node
I20260926 07:01:08.193554 21287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:08.193817 21287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:08.194981 21287 hybrid_clock.cc:648] HybridClock initialized: now 1790406068194975 us; error 35 us; skew 500 ppm
I20260926 07:01:08.196308 21287 webserver.cc:460] Webserver started at http://127.1.74.67:38039/ using document root <none> and password file <none>
I20260926 07:01:08.196574 21287 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:08.196645 21287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:08.196781 21287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:08.197849 21287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/data/instance:
uuid: "3f4b52a26cb5408e9b7d7ed87687586c"
format_stamp: "Formatted at 2026-09-26 07:01:08 on dist-test-slave-sfvt"
server_key: "3e7e732d4c477bbe611c41b879052d40"
server_key_iv: "56323ea4abf87478c3469fd9829caaae"
server_key_version: "encryptionkey@0"
I20260926 07:01:08.198223 21287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance:
uuid: "3f4b52a26cb5408e9b7d7ed87687586c"
format_stamp: "Formatted at 2026-09-26 07:01:08 on dist-test-slave-sfvt"
server_key: "3e7e732d4c477bbe611c41b879052d40"
server_key_iv: "56323ea4abf87478c3469fd9829caaae"
server_key_version: "encryptionkey@0"
I20260926 07:01:08.199630 21287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:08.200361 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:08.200574 21287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:08.200675 21287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/wal
uuid: "3f4b52a26cb5408e9b7d7ed87687586c"
format_stamp: "Formatted at 2026-09-26 07:01:08 on dist-test-slave-sfvt"
server_key: "3e7e732d4c477bbe611c41b879052d40"
server_key_iv: "56323ea4abf87478c3469fd9829caaae"
server_key_version: "encryptionkey@0"
I20260926 07:01:08.200758 21287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:08.220486 21287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:08.220770 21287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:08.220875 21287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:08.221055 21287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:08.221390 21287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:08.221429 21287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:08.221487 21287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:08.221514 21287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:08.227075 21287 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.67:38923
I20260926 07:01:08.227160 21414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.67:38923 every 8 connection(s)
I20260926 07:01:08.227514 21287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/data/info.pb
I20260926 07:01:08.232606 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.228320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.67:57341 (local address 127.1.74.126:33857)
0926 07:01:08.228406 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:08.228408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:08.228599 (+ 191us) server_negotiation.cc:407] Connection header received
0926 07:01:08.229080 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:08.229081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:08.229092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:08.229142 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:08.229774 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.230277 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.231239 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.231489 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.231542 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.232407 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:08.232408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:08.232408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:08.232418 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:08.232428 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:08.232431 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:08.232467 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:08.232529 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 07:01:08.232547 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:01:08.232802 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.228284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:33857 (local address 127.1.74.67:57341)
0926 07:01:08.228483 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:08.228517 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:08.228996 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:08.229203 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:08.229206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:08.229215 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:08.229706 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.229709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.230380 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.230380 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.231165 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.231167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.231563 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.231564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.231669 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.232247 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:01:08.232248 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:08.232277 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:01:08.232335 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:08.232469 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:08.232471 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:08.232473 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:08.232519 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 07:01:08.232593 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260926 07:01:08.233331 21415 heartbeater.cc:347] Connected to a master server at 127.1.74.126:33857
I20260926 07:01:08.233433 21415 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:08.233615 21415 heartbeater.cc:511] Master 127.1.74.126:33857 requested a full tablet report, sending...
I20260926 07:01:08.233994 20965 ts_manager.cc:194] Registered new tserver with Master: 3f4b52a26cb5408e9b7d7ed87687586c (127.1.74.67:38923)
I20260926 07:01:08.234345 20965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.74.67:57341
I20260926 07:01:08.236531 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 21287
I20260926 07:01:08.236629 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:08.236807 1321 external_mini_cluster.cc:1428] Setting key 14545907666d51944b366b92532f076a
I20260926 07:01:08.237650 1321 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:01:08.241999 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.239080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.1.74.126:33857)
0926 07:01:08.239172 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:08.239174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:08.239323 (+ 149us) server_negotiation.cc:407] Connection header received
0926 07:01:08.239397 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:08.239397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:08.239417 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:08.239465 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:08.239633 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.240102 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.240843 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.241048 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.241132 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.241746 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:08.241747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:08.241748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:08.241752 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:08.241765 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:08.241769 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:08.241834 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:08.241894 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:01:08.241916 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 07:01:08.243419 20965 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:08.245018 20965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:08.251367 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.247154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:38923 (local address 127.0.0.1:52526)
0926 07:01:08.247532 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:08.247544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:08.247598 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:08.247867 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:08.247870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:08.247872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:08.247940 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.247942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.249244 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.249244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.250138 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.250140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.251014 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.251015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.251092 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.251107 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 07:01:08.251190 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:01:08.251279 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260926 07:01:08.251382 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.247361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.1.74.67:38923)
0926 07:01:08.247553 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:08.247558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:08.247593 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:01:08.247667 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:08.247670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:08.247686 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:08.247766 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:08.248025 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.249153 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.250220 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.250925 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.251004 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.251098 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:08.251214 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 07:01:08.251276 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
I20260926 07:01:08.251484 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.247154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:33081 (local address 127.0.0.1:53468)
0926 07:01:08.247316 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:08.247376 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:08.247448 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:08.248130 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:08.248132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:08.248133 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:08.248168 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.248169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.249383 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.249384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.250496 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.250498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.251248 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.251249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.251327 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.251331 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:08.251362 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 07:01:08.251415 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260926 07:01:08.251565 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.247583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.1.74.65:42797)
0926 07:01:08.247790 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:08.247795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:08.247826 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:01:08.247957 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:08.247960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:08.247976 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:08.248053 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:08.248192 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.248889 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.250297 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.250998 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.251055 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.251126 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:08.251410 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:01:08.251457 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260926 07:01:08.251636 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.247377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:42797 (local address 127.0.0.1:56932)
0926 07:01:08.247768 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:08.247779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:08.247836 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:08.248066 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:08.248067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:08.248069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:08.248111 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.248113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.249005 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.249006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.250222 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.250224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.251188 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.251189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.251249 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.251254 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:08.251323 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:01:08.251364 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":31,"thread_start_us":73,"threads_started":1}
I20260926 07:01:08.252002 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.247350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.1.74.66:33081)
0926 07:01:08.247960 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:08.247964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:08.247974 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:08.248011 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:08.248013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:08.248028 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:08.248130 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:08.248242 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.249288 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.250581 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.251156 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.251636 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.251748 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:08.251847 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:01:08.251883 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":84,"threads_started":1}
I20260926 07:01:08.252610 21087 tablet_service.cc:1467] Processing CreateTablet for tablet ffc8d31576254146a5498737fbdac59e (DEFAULT_TABLE table=test-table [id=a235cb571f544b7c8a2acdcc931e5957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:08.252652 21218 tablet_service.cc:1467] Processing CreateTablet for tablet ffc8d31576254146a5498737fbdac59e (DEFAULT_TABLE table=test-table [id=a235cb571f544b7c8a2acdcc931e5957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:08.252988 21087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc8d31576254146a5498737fbdac59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:08.252987 21218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc8d31576254146a5498737fbdac59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:08.256407 21429 tablet_bootstrap.cc:492] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af: Bootstrap starting.
I20260926 07:01:08.257108 21429 tablet_bootstrap.cc:654] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:08.257016 21349 tablet_service.cc:1467] Processing CreateTablet for tablet ffc8d31576254146a5498737fbdac59e (DEFAULT_TABLE table=test-table [id=a235cb571f544b7c8a2acdcc931e5957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:08.257333 21349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc8d31576254146a5498737fbdac59e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:08.257444 21429 log.cc:824] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:08.257779 21430 tablet_bootstrap.cc:492] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a: Bootstrap starting.
I20260926 07:01:08.258361 21430 tablet_bootstrap.cc:654] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:08.258701 21430 log.cc:824] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:08.259244 21429 tablet_bootstrap.cc:492] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af: No bootstrap required, opened a new log
I20260926 07:01:08.259364 21429 ts_tablet_manager.cc:1397] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:01:08.259495 21430 tablet_bootstrap.cc:492] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a: No bootstrap required, opened a new log
I20260926 07:01:08.259570 21430 ts_tablet_manager.cc:1397] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:08.260743 21433 tablet_bootstrap.cc:492] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: Bootstrap starting.
I20260926 07:01:08.261408 21433 tablet_bootstrap.cc:654] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:08.261724 21433 log.cc:824] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:08.261731 21429 raft_consensus.cc:348] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } }
I20260926 07:01:08.261914 21429 raft_consensus.cc:374] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:08.261770 21430 raft_consensus.cc:348] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } }
I20260926 07:01:08.261968 21430 raft_consensus.cc:374] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:08.261996 21429 raft_consensus.cc:729] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2730f47b54cf4d26933e1b7cc471a6af, State: Initialized, Role: FOLLOWER
I20260926 07:01:08.262028 21430 raft_consensus.cc:729] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef89e9c5ce942d6b09d50d14eb0d98a, State: Initialized, Role: FOLLOWER
I20260926 07:01:08.262163 21429 consensus_queue.cc:260] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } }
I20260926 07:01:08.262176 21430 consensus_queue.cc:260] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } }
I20260926 07:01:08.262442 21433 tablet_bootstrap.cc:492] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: No bootstrap required, opened a new log
I20260926 07:01:08.262447 21430 ts_tablet_manager.cc:1428] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:01:08.262504 21433 ts_tablet_manager.cc:1397] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:08.262507 21284 heartbeater.cc:503] Master 127.1.74.126:33857 was elected leader, sending a full tablet report...
I20260926 07:01:08.262451 21429 ts_tablet_manager.cc:1428] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:01:08.262893 21153 heartbeater.cc:503] Master 127.1.74.126:33857 was elected leader, sending a full tablet report...
I20260926 07:01:08.264714 21433 raft_consensus.cc:348] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } }
I20260926 07:01:08.264842 21433 raft_consensus.cc:374] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:08.264864 21433 raft_consensus.cc:729] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4b52a26cb5408e9b7d7ed87687586c, State: Initialized, Role: FOLLOWER
I20260926 07:01:08.264978 21433 consensus_queue.cc:260] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } }
I20260926 07:01:08.265197 21433 ts_tablet_manager.cc:1428] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:01:08.265275 21415 heartbeater.cc:503] Master 127.1.74.126:33857 was elected leader, sending a full tablet report...
I20260926 07:01:08.284224 21435 raft_consensus.cc:482] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:08.284353 21435 raft_consensus.cc:504] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } }
I20260926 07:01:08.284677 21435 leader_election.cc:288] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f4b52a26cb5408e9b7d7ed87687586c (127.1.74.67:38923), 2730f47b54cf4d26933e1b7cc471a6af (127.1.74.65:42797)
I20260926 07:01:08.288601 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.284954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:57895 (local address 127.1.74.67:38923)
0926 07:01:08.284998 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:08.285000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:08.285009 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:08.285041 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:08.285042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:08.285052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:08.285144 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:08.285343 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.286106 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.287684 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.288237 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.288332 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.288341 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:08.288490 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 07:01:08.288530 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:01:08.288601 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.284952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.74.66:49247 (local address 127.1.74.65:42797)
0926 07:01:08.285042 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:08.285044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:08.285118 (+ 74us) server_negotiation.cc:407] Connection header received
0926 07:01:08.285176 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:08.285178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:08.285190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:08.285239 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:08.285401 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.286129 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.287685 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.288237 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.288315 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.288327 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:08.288490 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:01:08.288530 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 07:01:08.288617 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.284824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.67:38923 (local address 127.1.74.66:57895)
0926 07:01:08.284894 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:08.284907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:08.284950 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:08.285158 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:08.285159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:08.285161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:08.285258 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.285259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.286202 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.286202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.287588 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.287591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.288327 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.288328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.288412 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.288415 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:08.288465 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:01:08.288517 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":61}
I20260926 07:01:08.288623 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.284872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:42797 (local address 127.1.74.66:49247)
0926 07:01:08.285038 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:08.285051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:08.285101 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:08.285263 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:08.285264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:08.285266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:08.285329 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.285331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.286203 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.286203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.287588 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:08.287591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.288340 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.288341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:08.288401 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.288406 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:08.288466 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:01:08.288517 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":62,"thread_start_us":109,"threads_started":1}
I20260926 07:01:08.289021 21369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc8d31576254146a5498737fbdac59e" candidate_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" is_pre_election: true
I20260926 07:01:08.289178 21369 raft_consensus.cc:2454] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef89e9c5ce942d6b09d50d14eb0d98a in term 0.
I20260926 07:01:08.289279 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc8d31576254146a5498737fbdac59e" candidate_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2730f47b54cf4d26933e1b7cc471a6af" is_pre_election: true
I20260926 07:01:08.289448 21107 raft_consensus.cc:2454] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef89e9c5ce942d6b09d50d14eb0d98a in term 0.
I20260926 07:01:08.289443 21174 leader_election.cc:302] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4b52a26cb5408e9b7d7ed87687586c, 7ef89e9c5ce942d6b09d50d14eb0d98a; no voters:
I20260926 07:01:08.289604 21435 raft_consensus.cc:2781] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:08.289676 21435 raft_consensus.cc:482] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:08.289707 21435 raft_consensus.cc:3037] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:08.290418 21435 raft_consensus.cc:504] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } }
I20260926 07:01:08.290567 21435 leader_election.cc:288] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [CANDIDATE]: Term 1 election: Requested vote from peers 3f4b52a26cb5408e9b7d7ed87687586c (127.1.74.67:38923), 2730f47b54cf4d26933e1b7cc471a6af (127.1.74.65:42797)
I20260926 07:01:08.290728 21369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc8d31576254146a5498737fbdac59e" candidate_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4b52a26cb5408e9b7d7ed87687586c"
I20260926 07:01:08.290758 21107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc8d31576254146a5498737fbdac59e" candidate_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2730f47b54cf4d26933e1b7cc471a6af"
I20260926 07:01:08.290807 21369 raft_consensus.cc:3037] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:08.290843 21107 raft_consensus.cc:3037] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:08.291597 21369 raft_consensus.cc:2454] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef89e9c5ce942d6b09d50d14eb0d98a in term 1.
I20260926 07:01:08.291698 21107 raft_consensus.cc:2454] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef89e9c5ce942d6b09d50d14eb0d98a in term 1.
I20260926 07:01:08.291898 21171 leader_election.cc:302] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2730f47b54cf4d26933e1b7cc471a6af, 7ef89e9c5ce942d6b09d50d14eb0d98a; no voters:
I20260926 07:01:08.291999 21435 raft_consensus.cc:2781] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:08.292177 21435 raft_consensus.cc:686] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 1 LEADER]: Becoming Leader. State: Replica: 7ef89e9c5ce942d6b09d50d14eb0d98a, State: Running, Role: LEADER
I20260926 07:01:08.292286 21435 consensus_queue.cc:237] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } }
I20260926 07:01:08.293123 20965 catalog_manager.cc:5495] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef89e9c5ce942d6b09d50d14eb0d98a (127.1.74.66). New cstate: current_term: 1 leader_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef89e9c5ce942d6b09d50d14eb0d98a" member_type: VOTER last_known_addr { host: "127.1.74.66" port: 33081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 } health_report { overall_health: UNKNOWN } } }
I20260926 07:01:08.309036 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.306709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.1.74.66:33081)
0926 07:01:08.306774 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:08.306776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:08.306784 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:08.306838 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:08.306839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:08.306854 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:08.306914 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:08.307036 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.307588 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.308340 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:08.308518 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:08.308565 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:08.308696 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:08.308788 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:08.308849 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:08.308896 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 07:01:08.308911 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:01:08.310755 21369 raft_consensus.cc:1261] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [term 1 FOLLOWER]: Refusing update from remote peer 7ef89e9c5ce942d6b09d50d14eb0d98a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:08.310779 21107 raft_consensus.cc:1261] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [term 1 FOLLOWER]: Refusing update from remote peer 7ef89e9c5ce942d6b09d50d14eb0d98a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:08.311152 21439 consensus_queue.cc:1035] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2730f47b54cf4d26933e1b7cc471a6af" member_type: VOTER last_known_addr { host: "127.1.74.65" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:08.311324 21435 consensus_queue.cc:1035] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4b52a26cb5408e9b7d7ed87687586c" member_type: VOTER last_known_addr { host: "127.1.74.67" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:08.314254 21443 mvcc.cc:205] Tried to move back new op lower bound from 7333503255798972416 to 7333503255726403584. Current Snapshot: MvccSnapshot[applied={T|T < 7333503255798972416}]
I20260926 07:01:08.314683 21444 mvcc.cc:205] Tried to move back new op lower bound from 7333503255798972416 to 7333503255726403584. Current Snapshot: MvccSnapshot[applied={T|T < 7333503255798972416}]
I20260926 07:01:08.316579 20965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:08.316666 20965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:08.317466 20965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0304a83e5bb6446c9a57787b8fe05a3c: Sending DeleteTablet for 3 replicas of tablet ffc8d31576254146a5498737fbdac59e
I20260926 07:01:08.317826 21087 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc8d31576254146a5498737fbdac59e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:08 UTC) from {username='slave'} at 127.0.0.1:56932
I20260926 07:01:08.317808 21349 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc8d31576254146a5498737fbdac59e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:08 UTC) from {username='slave'} at 127.0.0.1:52526
I20260926 07:01:08.318024 21453 tablet_replica.cc:330] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: stopping tablet replica
I20260926 07:01:08.318127 21453 raft_consensus.cc:2229] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:08.318239 21453 raft_consensus.cc:2258] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:08.318326 21218 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc8d31576254146a5498737fbdac59e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:08 UTC) from {username='slave'} at 127.0.0.1:53468
I20260926 07:01:08.318447 21452 tablet_replica.cc:330] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af: stopping tablet replica
I20260926 07:01:08.318468 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 21025
I20260926 07:01:08.318540 21452 raft_consensus.cc:2229] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:08.318634 21452 raft_consensus.cc:2258] T ffc8d31576254146a5498737fbdac59e P 2730f47b54cf4d26933e1b7cc471a6af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:08.321769 21453 ts_tablet_manager.cc:1905] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:08.323967 21453 ts_tablet_manager.cc:1918] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:08.324081 21453 log.cc:1196] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: Deleting WAL directory at /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790405833506994-1321-0/minicluster-data/ts-2/wal/wals/ffc8d31576254146a5498737fbdac59e
I20260926 07:01:08.324352 21453 ts_tablet_manager.cc:1939] T ffc8d31576254146a5498737fbdac59e P 3f4b52a26cb5408e9b7d7ed87687586c: Deleting consensus metadata
I20260926 07:01:08.323877 21454 tablet_replica.cc:330] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a: stopping tablet replica
I20260926 07:01:08.324862 21454 raft_consensus.cc:2229] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:01:08.324882 20952 catalog_manager.cc:4841] TS 3f4b52a26cb5408e9b7d7ed87687586c (127.1.74.67:38923): tablet ffc8d31576254146a5498737fbdac59e (table test-table [id=a235cb571f544b7c8a2acdcc931e5957]) successfully deleted
I20260926 07:01:08.324999 21454 raft_consensus.cc:2258] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:08.325469 21454 ts_tablet_manager.cc:1905] T ffc8d31576254146a5498737fbdac59e P 7ef89e9c5ce942d6b09d50d14eb0d98a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:01:08.326028 20949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.74.65:42797 (error 108)
I20260926 07:01:08.326259 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 21156
I20260926 07:01:08.327257 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.326419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.65:42797 (local address 127.0.0.1:56936)
0926 07:01:08.327134 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:08.327227 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.65:42797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":695}
W20260926 07:01:08.327811 20949 catalog_manager.cc:4584] TS 2730f47b54cf4d26933e1b7cc471a6af (127.1.74.65:42797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffc8d31576254146a5498737fbdac59e: Network error: Client connection negotiation failed: client connection to 127.1.74.65:42797: connect: Connection refused (error 111)
I20260926 07:01:08.330930 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:08.330816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.66:33081 (local address 127.0.0.1:53474)
0926 07:01:08.330876 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:08.330906 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.66:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 07:01:08.330991 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 21287
I20260926 07:01:08.336123 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 20934
2026-09-26T07:01:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (668 ms)
[----------] 6 tests from EncryptionPolicyTest (8185 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:08 dist-test-slave-sfvt krb5kdc[21458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:08 dist-test-slave-sfvt krb5kdc[21458](info): set up 2 sockets
Sep 26 07:01:08 dist-test-slave-sfvt krb5kdc[21458](info): commencing operation
krb5kdc: starting...
W20260926 07:01:09.399403 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:09 dist-test-slave-sfvt krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:09.518448 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:09.597765 21471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:09.597983 21471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:09.598006 21471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:09.599081 21471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:09.599115 21471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:09.599129 21471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:09.599143 21471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:09.599154 21471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:09.600526 21471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:09.600721 21471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:09.600896 21471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:09.605684 21479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:09.605805 21477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:09.605684 21476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:09.605913 21471 server_base.cc:1034] running on GCE node
I20260926 07:01:09.606194 21471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:09.606496 21471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:09.607656 21471 hybrid_clock.cc:648] HybridClock initialized: now 1790406069607642 us; error 43 us; skew 500 ppm
Sep 26 07:01:09 dist-test-slave-sfvt krb5kdc[21458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:09.609970 21471 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:09.610401 21471 webserver.cc:460] Webserver started at http://127.0.0.1:40661/ using document root <none> and password file <none>
I20260926 07:01:09.610625 21471 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:09.610688 21471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:09.610831 21471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:09.611934 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "9e53642cb15149af9cd76a2800682adf"
format_stamp: "Formatted at 2026-09-26 07:01:09 on dist-test-slave-sfvt"
server_key: "cf65ea3e9005a8158603440f6603a1d5"
server_key_iv: "37674bd00606ee38f8a2023d31888fac"
server_key_version: "encryptionkey@0"
I20260926 07:01:09.612298 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "9e53642cb15149af9cd76a2800682adf"
format_stamp: "Formatted at 2026-09-26 07:01:09 on dist-test-slave-sfvt"
server_key: "cf65ea3e9005a8158603440f6603a1d5"
server_key_iv: "37674bd00606ee38f8a2023d31888fac"
server_key_version: "encryptionkey@0"
I20260926 07:01:09.613704 21471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:09.614461 21486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:09.614674 21471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:09.614773 21471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "9e53642cb15149af9cd76a2800682adf"
format_stamp: "Formatted at 2026-09-26 07:01:09 on dist-test-slave-sfvt"
server_key: "cf65ea3e9005a8158603440f6603a1d5"
server_key_iv: "37674bd00606ee38f8a2023d31888fac"
server_key_version: "encryptionkey@0"
I20260926 07:01:09.614853 21471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:09.627758 21471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:09.629604 21471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:09.629782 21471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:09.633831 21471 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39115
I20260926 07:01:09.633881 21538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39115 every 8 connection(s)
I20260926 07:01:09.634248 21471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:09.635313 21539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:09.637579 21539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf: Bootstrap starting.
I20260926 07:01:09.638190 21539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:09.638473 21539 log.cc:824] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:09.639204 21539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf: No bootstrap required, opened a new log
I20260926 07:01:09.640784 21539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53642cb15149af9cd76a2800682adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39115 } }
I20260926 07:01:09.640923 21539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:09.640969 21539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e53642cb15149af9cd76a2800682adf, State: Initialized, Role: FOLLOWER
I20260926 07:01:09.641115 21539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53642cb15149af9cd76a2800682adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39115 } }
I20260926 07:01:09.641181 21539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:09.641222 21539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:09.641271 21539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:09.641782 21539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53642cb15149af9cd76a2800682adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39115 } }
I20260926 07:01:09.641891 21539 leader_election.cc:302] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e53642cb15149af9cd76a2800682adf; no voters:
I20260926 07:01:09.642061 21539 leader_election.cc:288] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:09.642117 21542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:09.642330 21542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [term 1 LEADER]: Becoming Leader. State: Replica: 9e53642cb15149af9cd76a2800682adf, State: Running, Role: LEADER
I20260926 07:01:09.642315 21539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:09.642473 21542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53642cb15149af9cd76a2800682adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39115 } }
I20260926 07:01:09.643121 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 21471
I20260926 07:01:09.643240 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:09.643476 1321 external_mini_cluster.cc:1428] Setting key e54fc014ba2f823fac296e254c298bff
I20260926 07:01:09.644132 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e53642cb15149af9cd76a2800682adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53642cb15149af9cd76a2800682adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39115 } } }
I20260926 07:01:09.644199 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:09.644389 21554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:09.643868 21544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e53642cb15149af9cd76a2800682adf. Latest consensus state: current_term: 1 leader_uuid: "9e53642cb15149af9cd76a2800682adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e53642cb15149af9cd76a2800682adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39115 } } }
I20260926 07:01:09.644510 21544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:09.645220 21554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:09.646817 21554 catalog_manager.cc:1347] Generated new cluster ID: 9aaae63ef9f34ca9b76838fe39273da3
I20260926 07:01:09.646867 21554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:09 dist-test-slave-sfvt krb5kdc[21458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:01:09.653007 21554 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:09.653604 21554 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:09.656060 21554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e53642cb15149af9cd76a2800682adf: Generated new TSK 0
I20260926 07:01:09.656226 21554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:09.656745 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:09.644131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53207 (local address 127.0.0.1:39115)
0926 07:01:09.644351 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:09.644360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:09.644393 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:01:09.644517 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:09.644524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:09.644862 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:09.645035 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:09.645752 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:09.647180 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:09.648271 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:09.648888 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:09.648981 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:09.651915 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:09.651919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:09.651920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:09.651933 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:09.654671 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:09.655105 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:09.655109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:09.655109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:09.655230 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:09.655492 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:09.655493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:09.655493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:09.655793 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:09.655966 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:09.656421 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 07:01:09.656528 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
I20260926 07:01:09.657826 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:09.666280 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:09.659352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56881 (local address 127.0.0.1:39115)
0926 07:01:09.659386 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:09.659388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:09.659441 (+ 53us) server_negotiation.cc:407] Connection header received
0926 07:01:09.659510 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:09.659511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:09.659521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:09.659572 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:09.660249 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:09.661087 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:09.661789 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:09.662115 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:09.662184 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:09.663626 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:09.663628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:09.663628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:09.663635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:09.665094 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:09.665460 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:09.665461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:09.665462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:09.665519 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:09.665778 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:09.665779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:09.665779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:09.665906 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:09.666008 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:09.666107 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:01:09.666171 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:09.668171 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 21471
W20260926 07:01:09.672338 21456 connection.cc:535] client connection to 127.0.0.1:39115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39115): Connection reset by peer (error 104)
2026-09-26T07:01:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:09 dist-test-slave-sfvt krb5kdc[21570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:09 dist-test-slave-sfvt krb5kdc[21570](info): set up 2 sockets
Sep 26 07:01:09 dist-test-slave-sfvt krb5kdc[21570](info): commencing operation
krb5kdc: starting...
W20260926 07:01:10.695410 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:10 dist-test-slave-sfvt krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:10.826862 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:10.909089 21583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:10.909291 21583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:10.909315 21583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:10.910485 21583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:10.910530 21583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:10.910545 21583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:10.910581 21583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:10.910609 21583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:10.912141 21583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:10.912392 21583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:10.912611 21583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:01:10.917560 21583 server_base.cc:1034] running on GCE node
W20260926 07:01:10.917564 21588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:10.917601 21589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:10.917572 21591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:10.918074 21583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:10.918354 21583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:10.919489 21583 hybrid_clock.cc:648] HybridClock initialized: now 1790406070919480 us; error 30 us; skew 500 ppm
Sep 26 07:01:10 dist-test-slave-sfvt krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:10.929540 21583 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:10.930001 21583 webserver.cc:460] Webserver started at http://127.0.0.1:33115/ using document root <none> and password file <none>
I20260926 07:01:10.930241 21583 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:10.930287 21583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:10.930456 21583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:10.931622 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "b35ffc765cf74d02b27a7bcb91216d5e"
format_stamp: "Formatted at 2026-09-26 07:01:10 on dist-test-slave-sfvt"
server_key: "45579ef13df5b5800f50433fcb40156e"
server_key_iv: "e39706b56a1ce32d58bcd814256d8ae7"
server_key_version: "encryptionkey@0"
I20260926 07:01:10.931981 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "b35ffc765cf74d02b27a7bcb91216d5e"
format_stamp: "Formatted at 2026-09-26 07:01:10 on dist-test-slave-sfvt"
server_key: "45579ef13df5b5800f50433fcb40156e"
server_key_iv: "e39706b56a1ce32d58bcd814256d8ae7"
server_key_version: "encryptionkey@0"
I20260926 07:01:10.933300 21583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:10.934024 21598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:10.934195 21583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:10.934300 21583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "b35ffc765cf74d02b27a7bcb91216d5e"
format_stamp: "Formatted at 2026-09-26 07:01:10 on dist-test-slave-sfvt"
server_key: "45579ef13df5b5800f50433fcb40156e"
server_key_iv: "e39706b56a1ce32d58bcd814256d8ae7"
server_key_version: "encryptionkey@0"
I20260926 07:01:10.934372 21583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:10.952450 21583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:10.954394 21583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:10.954507 21583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:10.958653 21583 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36401
I20260926 07:01:10.958688 21650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36401 every 8 connection(s)
I20260926 07:01:10.959053 21583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:10.960125 21651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:10.962088 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 21583
I20260926 07:01:10.962208 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:10.962407 1321 external_mini_cluster.cc:1428] Setting key 6f7db4db17df9faa257a6915e16a3f44
I20260926 07:01:10.962663 21651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e: Bootstrap starting.
I20260926 07:01:10.963523 21651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:10.963831 21651 log.cc:824] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:10.964529 21651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e: No bootstrap required, opened a new log
I20260926 07:01:10.966279 21651 raft_consensus.cc:348] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35ffc765cf74d02b27a7bcb91216d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36401 } }
I20260926 07:01:10.966440 21651 raft_consensus.cc:374] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:10.966529 21651 raft_consensus.cc:729] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35ffc765cf74d02b27a7bcb91216d5e, State: Initialized, Role: FOLLOWER
I20260926 07:01:10.966702 21651 consensus_queue.cc:260] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35ffc765cf74d02b27a7bcb91216d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36401 } }
I20260926 07:01:10.966807 21651 raft_consensus.cc:388] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:10.966864 21651 raft_consensus.cc:482] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:10.966915 21651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:10.967535 21651 raft_consensus.cc:504] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35ffc765cf74d02b27a7bcb91216d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36401 } }
I20260926 07:01:10.967661 21651 leader_election.cc:302] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b35ffc765cf74d02b27a7bcb91216d5e; no voters:
I20260926 07:01:10.967851 21651 leader_election.cc:288] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:10.967886 21656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:10.968107 21656 raft_consensus.cc:686] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [term 1 LEADER]: Becoming Leader. State: Replica: b35ffc765cf74d02b27a7bcb91216d5e, State: Running, Role: LEADER
I20260926 07:01:10.968115 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:10.963012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48961 (local address 127.0.0.1:36401)
0926 07:01:10.963203 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:10.963211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:10.963310 (+ 99us) server_negotiation.cc:407] Connection header received
0926 07:01:10.963858 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:10.963866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:10.963934 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:10.964092 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:10.964277 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:10.965233 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:10.966016 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:10.966427 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:10.966568 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:10.967389 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:10.967393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:10.967393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:10.967576 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:10.967608 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:10.967619 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:10.967714 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:10.967864 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 07:01:10.967895 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260926 07:01:10.968180 21651 sys_catalog.cc:564] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:10.968250 21656 consensus_queue.cc:237] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35ffc765cf74d02b27a7bcb91216d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36401 } }
I20260926 07:01:10.968645 21657 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b35ffc765cf74d02b27a7bcb91216d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35ffc765cf74d02b27a7bcb91216d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36401 } } }
I20260926 07:01:10.968672 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b35ffc765cf74d02b27a7bcb91216d5e. Latest consensus state: current_term: 1 leader_uuid: "b35ffc765cf74d02b27a7bcb91216d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35ffc765cf74d02b27a7bcb91216d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36401 } } }
I20260926 07:01:10.968719 21657 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:10.968767 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:10.968961 21663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:10.969965 21663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:10.970002 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:10.972177 21663 catalog_manager.cc:1347] Generated new cluster ID: cd3fb5485d94403392de4bd204372bb5
I20260926 07:01:10.972231 21663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:10.974640 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:10.971591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35139 (local address 127.0.0.1:36401)
0926 07:01:10.971630 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:10.971633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:10.971728 (+ 95us) server_negotiation.cc:407] Connection header received
0926 07:01:10.971793 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:10.971794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:10.971804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:10.971869 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:10.972017 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:10.972620 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:10.973358 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:10.973597 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:10.973675 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:10.974376 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:10.974378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:10.974378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:10.974382 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:10.974397 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:10.974401 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:10.974454 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:10.974514 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:01:10.974523 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:10.981560 21663 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:10.982234 21663 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:10.987991 21663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b35ffc765cf74d02b27a7bcb91216d5e: Generated new TSK 0
I20260926 07:01:10.988174 21663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:10.993435 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 21583
W20260926 07:01:10.997779 21568 connection.cc:535] client connection to 127.0.0.1:36401 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36401): Connection reset by peer (error 104)
2026-09-26T07:01:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:11 dist-test-slave-sfvt krb5kdc[21682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:11 dist-test-slave-sfvt krb5kdc[21682](info): set up 2 sockets
Sep 26 07:01:11 dist-test-slave-sfvt krb5kdc[21682](info): commencing operation
krb5kdc: starting...
W20260926 07:01:12.035301 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:12 dist-test-slave-sfvt krb5kdc[21682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:12.154352 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:12.236769 21695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:12.237030 21695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:12.237068 21695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:12.238238 21695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:12.238312 21695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:12.238337 21695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:12.238373 21695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:12.238402 21695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:12.239923 21695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:12.240211 21695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:12.240449 21695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:12.245479 21703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:12.245479 21701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:12.245543 21700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:12.245613 21695 server_base.cc:1034] running on GCE node
I20260926 07:01:12.245950 21695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:12.246271 21695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:12.247440 21695 hybrid_clock.cc:648] HybridClock initialized: now 1790406072247429 us; error 47 us; skew 500 ppm
Sep 26 07:01:12 dist-test-slave-sfvt krb5kdc[21682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:12.249816 21695 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:12.250196 21695 webserver.cc:460] Webserver started at http://127.0.0.1:36239/ using document root <none> and password file <none>
I20260926 07:01:12.250373 21695 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:12.250413 21695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:12.250519 21695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:12.251614 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "60f869bed95a41a89e76f64642a68c3a"
format_stamp: "Formatted at 2026-09-26 07:01:12 on dist-test-slave-sfvt"
server_key: "e695e63e056a4894b423858c8f6ce9f7"
server_key_iv: "9abf73698221b525cd80e325a571f985"
server_key_version: "encryptionkey@0"
I20260926 07:01:12.251910 21695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "60f869bed95a41a89e76f64642a68c3a"
format_stamp: "Formatted at 2026-09-26 07:01:12 on dist-test-slave-sfvt"
server_key: "e695e63e056a4894b423858c8f6ce9f7"
server_key_iv: "9abf73698221b525cd80e325a571f985"
server_key_version: "encryptionkey@0"
I20260926 07:01:12.253121 21695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:12.253803 21710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:12.253998 21695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:12.254079 21695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "60f869bed95a41a89e76f64642a68c3a"
format_stamp: "Formatted at 2026-09-26 07:01:12 on dist-test-slave-sfvt"
server_key: "e695e63e056a4894b423858c8f6ce9f7"
server_key_iv: "9abf73698221b525cd80e325a571f985"
server_key_version: "encryptionkey@0"
I20260926 07:01:12.254133 21695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:12.257314 21695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:12.259178 21695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:12.259358 21695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:12.263056 21695 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34667
I20260926 07:01:12.263084 21762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34667 every 8 connection(s)
I20260926 07:01:12.263481 21695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:12.264482 21763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:12.266863 21763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a: Bootstrap starting.
I20260926 07:01:12.267490 21763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:12.267769 21763 log.cc:824] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:12.268460 21763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a: No bootstrap required, opened a new log
I20260926 07:01:12.268522 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 21695
I20260926 07:01:12.268668 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:12.268877 1321 external_mini_cluster.cc:1428] Setting key ccbfcc142f4062be9e09afa6a546c3dd
I20260926 07:01:12.270444 21763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f869bed95a41a89e76f64642a68c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34667 } }
I20260926 07:01:12.270627 21763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:12.270669 21763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f869bed95a41a89e76f64642a68c3a, State: Initialized, Role: FOLLOWER
I20260926 07:01:12.270857 21763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f869bed95a41a89e76f64642a68c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34667 } }
I20260926 07:01:12.270968 21763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:12.271018 21763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:12.271072 21763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:12.271822 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:12.269368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34961 (local address 127.0.0.1:34667)
0926 07:01:12.269541 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:12.269549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:12.269574 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:01:12.270422 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:12.270429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:12.270454 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:12.270567 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:12.271331 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:12.271335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:12.271343 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:12.271347 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:12.271390 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:12.271401 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:12.271456 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:12.271558 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:01:12.271608 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260926 07:01:12.272264 21763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f869bed95a41a89e76f64642a68c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34667 } }
I20260926 07:01:12.272369 21763 leader_election.cc:302] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60f869bed95a41a89e76f64642a68c3a; no voters:
I20260926 07:01:12.272526 21763 leader_election.cc:288] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:12.272558 21768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:12.272732 21768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [term 1 LEADER]: Becoming Leader. State: Replica: 60f869bed95a41a89e76f64642a68c3a, State: Running, Role: LEADER
I20260926 07:01:12.272766 21763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:12.272837 21768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f869bed95a41a89e76f64642a68c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34667 } }
I20260926 07:01:12.272917 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:12.273155 21768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60f869bed95a41a89e76f64642a68c3a. Latest consensus state: current_term: 1 leader_uuid: "60f869bed95a41a89e76f64642a68c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f869bed95a41a89e76f64642a68c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34667 } } }
I20260926 07:01:12.273168 21770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60f869bed95a41a89e76f64642a68c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f869bed95a41a89e76f64642a68c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34667 } } }
I20260926 07:01:12.273242 21768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:12.273245 21770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a [sys.catalog]: This master's current role is: LEADER
W20260926 07:01:12.274606 21786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:12.274668 21786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:12.274771 21775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:12.274947 21775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:12.275823 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:12.274540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44229 (local address 127.0.0.1:34667)
0926 07:01:12.274572 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:12.274574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:12.274811 (+ 237us) server_negotiation.cc:407] Connection header received
0926 07:01:12.274885 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:12.274886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:12.274892 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:12.274936 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:12.275610 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:12.275611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:12.275612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:12.275618 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:12.275633 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:12.275637 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:12.275701 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:12.275753 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 07:01:12.275766 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:12.276433 21775 catalog_manager.cc:1347] Generated new cluster ID: 918f4c5d5b59428eaf1fa8cbe74ce6aa
I20260926 07:01:12.276486 21775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:12.293730 21775 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:12.293987 21775 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:12.297143 21775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60f869bed95a41a89e76f64642a68c3a: Generated new TSK 0
I20260926 07:01:12.297355 21775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:12.301888 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 21695
2026-09-26T07:01:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:12 dist-test-slave-sfvt krb5kdc[21794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:12 dist-test-slave-sfvt krb5kdc[21794](info): set up 2 sockets
Sep 26 07:01:12 dist-test-slave-sfvt krb5kdc[21794](info): commencing operation
krb5kdc: starting...
W20260926 07:01:13.331408 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:13 dist-test-slave-sfvt krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:13.450955 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:13.531688 21807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:13.531953 21807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:13.531991 21807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:13.533133 21807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:13.533207 21807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:13.533233 21807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:13.533252 21807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:13.533283 21807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:13.534813 21807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:13.535104 21807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:13.535391 21807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:13.540328 21813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:13.540336 21812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:13.540372 21807 server_base.cc:1034] running on GCE node
W20260926 07:01:13.540364 21815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:13.540835 21807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:13.541132 21807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:13.542277 21807 hybrid_clock.cc:648] HybridClock initialized: now 1790406073542260 us; error 37 us; skew 500 ppm
Sep 26 07:01:13 dist-test-slave-sfvt krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:13.544698 21807 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:13.545212 21807 webserver.cc:460] Webserver started at http://127.0.0.1:43745/ using document root <none> and password file <none>
I20260926 07:01:13.545439 21807 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:13.545508 21807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:13.545658 21807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:13.546728 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "1de965b4b66c436fa50faf3bccaad79d"
format_stamp: "Formatted at 2026-09-26 07:01:13 on dist-test-slave-sfvt"
server_key: "135325a8100bb7d6315d28ce61473c4a"
server_key_iv: "a11563c64d43110b2ee22d5be3eae115"
server_key_version: "encryptionkey@0"
I20260926 07:01:13.547087 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "1de965b4b66c436fa50faf3bccaad79d"
format_stamp: "Formatted at 2026-09-26 07:01:13 on dist-test-slave-sfvt"
server_key: "135325a8100bb7d6315d28ce61473c4a"
server_key_iv: "a11563c64d43110b2ee22d5be3eae115"
server_key_version: "encryptionkey@0"
I20260926 07:01:13.548463 21807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:13.549289 21822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:13.549481 21807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:13.549582 21807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "1de965b4b66c436fa50faf3bccaad79d"
format_stamp: "Formatted at 2026-09-26 07:01:13 on dist-test-slave-sfvt"
server_key: "135325a8100bb7d6315d28ce61473c4a"
server_key_iv: "a11563c64d43110b2ee22d5be3eae115"
server_key_version: "encryptionkey@0"
I20260926 07:01:13.549695 21807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:13.574664 21807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:13.576797 21807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:13.577005 21807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:13.581044 21807 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42173
I20260926 07:01:13.581084 21874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42173 every 8 connection(s)
I20260926 07:01:13.581461 21807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:13.582417 21875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:13.584813 21875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d: Bootstrap starting.
I20260926 07:01:13.585348 21875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:13.585577 21875 log.cc:824] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:13.586148 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 21807
I20260926 07:01:13.586268 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:13.586413 21875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d: No bootstrap required, opened a new log
I20260926 07:01:13.586472 1321 external_mini_cluster.cc:1428] Setting key 39790f823a219dfc1b7702e44b6d1660
I20260926 07:01:13.588884 21875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de965b4b66c436fa50faf3bccaad79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260926 07:01:13.589044 21875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:13.589090 21875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de965b4b66c436fa50faf3bccaad79d, State: Initialized, Role: FOLLOWER
I20260926 07:01:13.589222 21875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de965b4b66c436fa50faf3bccaad79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260926 07:01:13.589329 21875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:13.589372 21875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:13.589421 21875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:13.589949 21875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de965b4b66c436fa50faf3bccaad79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260926 07:01:13.590066 21875 leader_election.cc:302] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1de965b4b66c436fa50faf3bccaad79d; no voters:
I20260926 07:01:13.590245 21875 leader_election.cc:288] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:13.590270 21880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:13.590386 21880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [term 1 LEADER]: Becoming Leader. State: Replica: 1de965b4b66c436fa50faf3bccaad79d, State: Running, Role: LEADER
I20260926 07:01:13.590515 21875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:13.590518 21880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de965b4b66c436fa50faf3bccaad79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260926 07:01:13.590937 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de965b4b66c436fa50faf3bccaad79d. Latest consensus state: current_term: 1 leader_uuid: "1de965b4b66c436fa50faf3bccaad79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de965b4b66c436fa50faf3bccaad79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } } }
I20260926 07:01:13.590947 21881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1de965b4b66c436fa50faf3bccaad79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de965b4b66c436fa50faf3bccaad79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } } }
I20260926 07:01:13.591073 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:13.591158 21881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:13.591420 21889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:13.591933 21889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 07:01:13 dist-test-slave-sfvt krb5kdc[21794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:01:13.594550 21889 catalog_manager.cc:1347] Generated new cluster ID: 509be5e829154909811087152ea3b64a
I20260926 07:01:13.594599 21889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:13.598404 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:13.587012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34687 (local address 127.0.0.1:42173)
0926 07:01:13.587180 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:13.587189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:13.587217 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:01:13.587833 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:13.587840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:13.587890 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:13.588002 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:13.588785 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:13.590095 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:13.591113 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:13.591764 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:13.591848 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:13.594792 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:13.594796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:13.594796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:13.594808 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:13.596736 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:13.597170 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:13.597172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:13.597173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:13.597276 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:13.597503 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:13.597503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:13.597504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:13.597770 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:13.597911 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:13.598069 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:01:13.598204 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260926 07:01:13.599275 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:13.610864 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:13.603571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38617 (local address 127.0.0.1:42173)
0926 07:01:13.603614 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:13.603616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:13.603765 (+ 149us) server_negotiation.cc:407] Connection header received
0926 07:01:13.603812 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:13.603812 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:13.603822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:13.603875 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:13.604518 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:13.605261 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:13.606250 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:13.606548 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:13.606609 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:13.608069 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:13.608071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:13.608071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:13.608078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:13.609725 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:13.610117 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:13.610118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:13.610118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:13.610170 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:13.610368 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:13.610368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:13.610368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:13.610477 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:13.610582 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:13.610700 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:01:13.610771 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:13.626029 21889 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:13.626596 21889 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:13.633677 21889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1de965b4b66c436fa50faf3bccaad79d: Generated new TSK 0
I20260926 07:01:13.633894 21889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:13.636270 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 21807
2026-09-26T07:01:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:13 dist-test-slave-sfvt krb5kdc[21906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:13 dist-test-slave-sfvt krb5kdc[21906](info): set up 2 sockets
Sep 26 07:01:13 dist-test-slave-sfvt krb5kdc[21906](info): commencing operation
krb5kdc: starting...
W20260926 07:01:14.659354 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:14 dist-test-slave-sfvt krb5kdc[21906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:14.781804 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:14.865026 21919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:14.865273 21919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:14.865312 21919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:14.866447 21919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:14.866520 21919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:14.866544 21919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:14.866585 21919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:14.866609 21919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:14.868131 21919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:14.868410 21919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:14.868640 21919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:14.873509 21927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:14.873466 21925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:14.873468 21924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:14.873801 21919 server_base.cc:1034] running on GCE node
I20260926 07:01:14.874025 21919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:14.874262 21919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:14.875396 21919 hybrid_clock.cc:648] HybridClock initialized: now 1790406074875380 us; error 29 us; skew 500 ppm
Sep 26 07:01:14 dist-test-slave-sfvt krb5kdc[21906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:14.877908 21919 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:14.878283 21919 webserver.cc:460] Webserver started at http://127.0.0.1:34243/ using document root <none> and password file <none>
I20260926 07:01:14.878470 21919 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:14.878510 21919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:14.878594 21919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:14.879702 21919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "e281741dcc844285a69531cdd8438e90"
format_stamp: "Formatted at 2026-09-26 07:01:14 on dist-test-slave-sfvt"
server_key: "73aad3b37afed9669b0314f7f46c9bdc"
server_key_iv: "d89e4d832982c692ec04c2b07618c29f"
server_key_version: "encryptionkey@0"
I20260926 07:01:14.880012 21919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "e281741dcc844285a69531cdd8438e90"
format_stamp: "Formatted at 2026-09-26 07:01:14 on dist-test-slave-sfvt"
server_key: "73aad3b37afed9669b0314f7f46c9bdc"
server_key_iv: "d89e4d832982c692ec04c2b07618c29f"
server_key_version: "encryptionkey@0"
I20260926 07:01:14.881223 21919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:14.881884 21934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:14.882048 21919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:14.882159 21919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "e281741dcc844285a69531cdd8438e90"
format_stamp: "Formatted at 2026-09-26 07:01:14 on dist-test-slave-sfvt"
server_key: "73aad3b37afed9669b0314f7f46c9bdc"
server_key_iv: "d89e4d832982c692ec04c2b07618c29f"
server_key_version: "encryptionkey@0"
I20260926 07:01:14.882237 21919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:14.898023 21919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:14.900780 21919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:14.900928 21919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:14.905748 21919 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40841
I20260926 07:01:14.905797 21986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40841 every 8 connection(s)
I20260926 07:01:14.906199 21919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:14.907155 21987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:14.909431 21987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90: Bootstrap starting.
I20260926 07:01:14.909966 21987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:14.910198 21987 log.cc:824] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:14.910849 21987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90: No bootstrap required, opened a new log
I20260926 07:01:14.912484 21987 raft_consensus.cc:348] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e281741dcc844285a69531cdd8438e90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40841 } }
I20260926 07:01:14.912659 21987 raft_consensus.cc:374] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:14.912683 21987 raft_consensus.cc:729] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e281741dcc844285a69531cdd8438e90, State: Initialized, Role: FOLLOWER
I20260926 07:01:14.912792 21987 consensus_queue.cc:260] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e281741dcc844285a69531cdd8438e90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40841 } }
I20260926 07:01:14.912880 21987 raft_consensus.cc:388] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:14.912917 21987 raft_consensus.cc:482] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:14.912992 21987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:14.913555 21987 raft_consensus.cc:504] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e281741dcc844285a69531cdd8438e90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40841 } }
I20260926 07:01:14.913677 21987 leader_election.cc:302] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e281741dcc844285a69531cdd8438e90; no voters:
I20260926 07:01:14.913867 21987 leader_election.cc:288] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:14.913919 21990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:14.914126 21987 sys_catalog.cc:564] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:14.914140 21990 raft_consensus.cc:686] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [term 1 LEADER]: Becoming Leader. State: Replica: e281741dcc844285a69531cdd8438e90, State: Running, Role: LEADER
I20260926 07:01:14.914335 21990 consensus_queue.cc:237] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e281741dcc844285a69531cdd8438e90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40841 } }
I20260926 07:01:14.914955 21991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e281741dcc844285a69531cdd8438e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e281741dcc844285a69531cdd8438e90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40841 } } }
I20260926 07:01:14.914966 21992 sys_catalog.cc:455] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e281741dcc844285a69531cdd8438e90. Latest consensus state: current_term: 1 leader_uuid: "e281741dcc844285a69531cdd8438e90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e281741dcc844285a69531cdd8438e90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40841 } } }
I20260926 07:01:14.915067 21991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:14.915073 21992 sys_catalog.cc:458] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:14.915292 21999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:14.915793 21999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:14.916335 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 21919
I20260926 07:01:14.916472 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:14.916678 1321 external_mini_cluster.cc:1428] Setting key 5980f99950d4f34cb1293eddde46b1f6
I20260926 07:01:14.917515 21999 catalog_manager.cc:1347] Generated new cluster ID: 2938a4e4cffe49d2b125da62fdc61bfb
I20260926 07:01:14.917572 21999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:14.922875 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:14.917322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53611 (local address 127.0.0.1:40841)
0926 07:01:14.917486 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:14.917495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:14.917525 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:01:14.917646 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:14.917654 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:14.917710 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:14.917843 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:14.918090 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:14.919470 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:14.920585 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:14.921190 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:14.921293 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:14.922013 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:14.922017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:14.922017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:14.922226 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:14.922262 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:14.922276 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:14.922334 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:14.922435 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 07:01:14.922496 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260926 07:01:14.923823 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:14.928689 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:14.925351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58929 (local address 127.0.0.1:40841)
0926 07:01:14.925384 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:14.925386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:14.925417 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:01:14.925437 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:14.925437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:14.925447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:14.925497 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:14.925630 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:14.926341 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:14.927324 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:14.927636 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:14.927707 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:14.928358 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:14.928360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:14.928360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:14.928365 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:14.928381 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:14.928385 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:14.928444 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:14.928496 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 07:01:14.928519 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:14.932380 21999 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:14.933002 21999 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:14.935676 21999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e281741dcc844285a69531cdd8438e90: Generated new TSK 0
I20260926 07:01:14.935854 21999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:14.940868 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 21919
2026-09-26T07:01:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:14 dist-test-slave-sfvt krb5kdc[22018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:14 dist-test-slave-sfvt krb5kdc[22018](info): set up 2 sockets
Sep 26 07:01:14 dist-test-slave-sfvt krb5kdc[22018](info): commencing operation
krb5kdc: starting...
W20260926 07:01:15.963425 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:16 dist-test-slave-sfvt krb5kdc[22018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:16.081610 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:16.163132 22031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:16.163442 22031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:16.163484 22031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:16.164700 22031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:16.164772 22031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:16.164796 22031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:16.164829 22031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:16.164857 22031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:16.166360 22031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:16.166637 22031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:16.166867 22031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:16.172214 22037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:16.172272 22031 server_base.cc:1034] running on GCE node
W20260926 07:01:16.172205 22039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:16.172230 22036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:16.172646 22031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:16.172895 22031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:16.174126 22031 hybrid_clock.cc:648] HybridClock initialized: now 1790406076174114 us; error 47 us; skew 500 ppm
Sep 26 07:01:16 dist-test-slave-sfvt krb5kdc[22018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:16.181061 22031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:16.181547 22031 webserver.cc:460] Webserver started at http://127.0.0.1:37075/ using document root <none> and password file <none>
I20260926 07:01:16.181800 22031 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:16.181846 22031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:16.182024 22031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:16.183095 22031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "2d75e2009f9840bf9a39bf8c8b3fde69"
format_stamp: "Formatted at 2026-09-26 07:01:16 on dist-test-slave-sfvt"
server_key: "31a86720daa410d03157afe8372dc96e"
server_key_iv: "92c7f3aa46962b02386b627615afa896"
server_key_version: "encryptionkey@0"
I20260926 07:01:16.183476 22031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "2d75e2009f9840bf9a39bf8c8b3fde69"
format_stamp: "Formatted at 2026-09-26 07:01:16 on dist-test-slave-sfvt"
server_key: "31a86720daa410d03157afe8372dc96e"
server_key_iv: "92c7f3aa46962b02386b627615afa896"
server_key_version: "encryptionkey@0"
I20260926 07:01:16.184731 22031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:16.185413 22046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:16.185585 22031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:16.185684 22031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "2d75e2009f9840bf9a39bf8c8b3fde69"
format_stamp: "Formatted at 2026-09-26 07:01:16 on dist-test-slave-sfvt"
server_key: "31a86720daa410d03157afe8372dc96e"
server_key_iv: "92c7f3aa46962b02386b627615afa896"
server_key_version: "encryptionkey@0"
I20260926 07:01:16.185770 22031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:16.189109 22031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:16.190958 22031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:16.191114 22031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:16.195062 22031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45513
I20260926 07:01:16.195092 22098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45513 every 8 connection(s)
I20260926 07:01:16.195477 22031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:16.196260 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 22031
I20260926 07:01:16.196379 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:16.196573 1321 external_mini_cluster.cc:1428] Setting key 1b824d0af08e3afa1b7d85c21d07e344
I20260926 07:01:16.196596 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:16.198932 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:16.197093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44823 (local address 127.0.0.1:45513)
0926 07:01:16.197286 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:16.197293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:16.197317 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:01:16.197707 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:16.197713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:16.197738 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:16.197851 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:16.198489 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:16.198492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:16.198493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:16.198496 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:16.198533 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:16.198543 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:16.198597 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:16.198683 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 07:01:16.198738 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260926 07:01:16.200237 22099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69: Bootstrap starting.
I20260926 07:01:16.200392 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:16.200960 22099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:16.201303 22099 log.cc:824] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:16.202091 22099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69: No bootstrap required, opened a new log
I20260926 07:01:16.203409 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:16.202017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34085 (local address 127.0.0.1:45513)
0926 07:01:16.202267 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:16.202269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:16.202276 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:01:16.202296 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:16.202297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:16.202301 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:16.202346 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:16.203136 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:16.203137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:16.203137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:16.203143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:16.203158 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:16.203162 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:16.203227 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:16.203322 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:01:16.203335 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260926 07:01:16.204564 22099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75e2009f9840bf9a39bf8c8b3fde69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } }
I20260926 07:01:16.204700 22099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:16.204722 22099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d75e2009f9840bf9a39bf8c8b3fde69, State: Initialized, Role: FOLLOWER
I20260926 07:01:16.204864 22099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75e2009f9840bf9a39bf8c8b3fde69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } }
I20260926 07:01:16.204960 22099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:16.205024 22099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:16.205066 22099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:16.205637 22099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75e2009f9840bf9a39bf8c8b3fde69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } }
I20260926 07:01:16.205747 22099 leader_election.cc:302] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d75e2009f9840bf9a39bf8c8b3fde69; no voters:
I20260926 07:01:16.205950 22099 leader_election.cc:288] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:16.205987 22109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:16.206140 22109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [term 1 LEADER]: Becoming Leader. State: Replica: 2d75e2009f9840bf9a39bf8c8b3fde69, State: Running, Role: LEADER
I20260926 07:01:16.206192 22099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:16.206279 22109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75e2009f9840bf9a39bf8c8b3fde69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } }
I20260926 07:01:16.206709 22111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d75e2009f9840bf9a39bf8c8b3fde69. Latest consensus state: current_term: 1 leader_uuid: "2d75e2009f9840bf9a39bf8c8b3fde69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75e2009f9840bf9a39bf8c8b3fde69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } } }
I20260926 07:01:16.206696 22110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d75e2009f9840bf9a39bf8c8b3fde69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75e2009f9840bf9a39bf8c8b3fde69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45513 } } }
I20260926 07:01:16.206936 22111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:16.206943 22110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:16.207451 22118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:16.207870 22118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:16.209379 22118 catalog_manager.cc:1347] Generated new cluster ID: c14af98e046343de8b6ce2044b6da898
I20260926 07:01:16.209430 22118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:16.218938 22118 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:16.219126 22118 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:16.226967 22118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d75e2009f9840bf9a39bf8c8b3fde69: Generated new TSK 0
I20260926 07:01:16.227174 22118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:16.234269 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 22031
2026-09-26T07:01:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:16 dist-test-slave-sfvt krb5kdc[22130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:16 dist-test-slave-sfvt krb5kdc[22130](info): set up 2 sockets
Sep 26 07:01:16 dist-test-slave-sfvt krb5kdc[22130](info): commencing operation
krb5kdc: starting...
W20260926 07:01:17.271402 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:17 dist-test-slave-sfvt krb5kdc[22130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:17.400090 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:17.480571 22143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:17.480779 22143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:17.480803 22143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:17.481856 22143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:17.481889 22143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:17.481904 22143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:17.481917 22143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:17.481930 22143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:17.483314 22143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:17.483500 22143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:17.483661 22143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:17.488493 22148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:17.488538 22149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:17.488504 22151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:17.488762 22143 server_base.cc:1034] running on GCE node
I20260926 07:01:17.488984 22143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:17.489226 22143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:17.490372 22143 hybrid_clock.cc:648] HybridClock initialized: now 1790406077490360 us; error 28 us; skew 500 ppm
Sep 26 07:01:17 dist-test-slave-sfvt krb5kdc[22130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:17.492811 22143 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:17.493238 22143 webserver.cc:460] Webserver started at http://127.0.0.1:43593/ using document root <none> and password file <none>
I20260926 07:01:17.493477 22143 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:17.493546 22143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:17.493698 22143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:17.494756 22143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "5962ec3e3b79457eacabfa6f22e1c237"
format_stamp: "Formatted at 2026-09-26 07:01:17 on dist-test-slave-sfvt"
server_key: "0d82e382df525629e6584abafb2adf56"
server_key_iv: "22a3fae8cf962fd1ca782f4428977387"
server_key_version: "encryptionkey@0"
I20260926 07:01:17.495129 22143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "5962ec3e3b79457eacabfa6f22e1c237"
format_stamp: "Formatted at 2026-09-26 07:01:17 on dist-test-slave-sfvt"
server_key: "0d82e382df525629e6584abafb2adf56"
server_key_iv: "22a3fae8cf962fd1ca782f4428977387"
server_key_version: "encryptionkey@0"
I20260926 07:01:17.496475 22143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:17.497165 22158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:17.497325 22143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:17.497422 22143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "5962ec3e3b79457eacabfa6f22e1c237"
format_stamp: "Formatted at 2026-09-26 07:01:17 on dist-test-slave-sfvt"
server_key: "0d82e382df525629e6584abafb2adf56"
server_key_iv: "22a3fae8cf962fd1ca782f4428977387"
server_key_version: "encryptionkey@0"
I20260926 07:01:17.497503 22143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:17.524295 22143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:17.526510 22143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:17.526713 22143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:17.530910 22143 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39831
I20260926 07:01:17.530994 22210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39831 every 8 connection(s)
I20260926 07:01:17.531445 22143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:17.532474 22211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:17.534776 22211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237: Bootstrap starting.
I20260926 07:01:17.534981 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 22143
I20260926 07:01:17.535102 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:17.535331 1321 external_mini_cluster.cc:1428] Setting key 27a8c9a8f5787c03cc726090d100f57c
I20260926 07:01:17.535537 22211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:17.535872 22211 log.cc:824] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:17.536607 22211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237: No bootstrap required, opened a new log
I20260926 07:01:17.538861 22211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962ec3e3b79457eacabfa6f22e1c237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260926 07:01:17.539027 22211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:17.539072 22211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5962ec3e3b79457eacabfa6f22e1c237, State: Initialized, Role: FOLLOWER
I20260926 07:01:17.539199 22211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962ec3e3b79457eacabfa6f22e1c237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260926 07:01:17.539345 22211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:17.539391 22211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:17.539443 22211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:17.540026 22211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962ec3e3b79457eacabfa6f22e1c237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260926 07:01:17.540125 22211 leader_election.cc:302] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5962ec3e3b79457eacabfa6f22e1c237; no voters:
I20260926 07:01:17.540292 22211 leader_election.cc:288] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:17.540346 22216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:17.540520 22216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [term 1 LEADER]: Becoming Leader. State: Replica: 5962ec3e3b79457eacabfa6f22e1c237, State: Running, Role: LEADER
I20260926 07:01:17.540525 22211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:17.540637 22216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962ec3e3b79457eacabfa6f22e1c237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260926 07:01:17.540987 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5962ec3e3b79457eacabfa6f22e1c237. Latest consensus state: current_term: 1 leader_uuid: "5962ec3e3b79457eacabfa6f22e1c237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962ec3e3b79457eacabfa6f22e1c237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260926 07:01:17.541000 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5962ec3e3b79457eacabfa6f22e1c237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5962ec3e3b79457eacabfa6f22e1c237" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260926 07:01:17.541139 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:17.541132 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:17.541538 22225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:17.542055 22225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 07:01:17 dist-test-slave-sfvt krb5kdc[22130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:01:17.543953 22225 catalog_manager.cc:1347] Generated new cluster ID: f5f7647971e941fc81e16a9cbdef9508
I20260926 07:01:17.544003 22225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:17.547322 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:17.535869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:41265 (local address 127.0.0.1:39831)
0926 07:01:17.536077 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:17.536086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:17.536118 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:01:17.536694 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:17.536704 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:17.536787 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:17.536928 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:17.537550 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:17.538971 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:17.539919 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:17.540532 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:17.540643 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:17.543600 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:17.543604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:17.543604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:17.543616 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:17.545659 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:17.546089 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:17.546093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:17.546093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:17.546195 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:17.546423 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:17.546424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:17.546424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:17.546736 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:17.546906 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:17.546984 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 07:01:17.547080 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260926 07:01:17.548323 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:17.558323 22225 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:17.558703 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:17.549700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:56223 (local address 127.0.0.1:39831)
0926 07:01:17.549731 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:17.549733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:17.551574 (+ 1841us) server_negotiation.cc:407] Connection header received
0926 07:01:17.551634 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:17.551634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:17.551644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:17.551697 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:17.552301 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:17.553023 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:17.554032 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:17.554321 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:17.554377 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:17.555804 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:17.555805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:17.555805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:17.555812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:17.557307 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:17.557709 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:17.557710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:17.557710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:17.557785 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:17.558022 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:17.558023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:17.558023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:17.558146 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:17.558275 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:17.558552 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:01:17.558618 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 07:01:17.559015 22225 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:17.562885 22225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5962ec3e3b79457eacabfa6f22e1c237: Generated new TSK 0
I20260926 07:01:17.563071 22225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:17.568063 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 22143
W20260926 07:01:17.572144 22128 connection.cc:535] client connection to 127.0.0.1:39831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39831): Connection reset by peer (error 104)
2026-09-26T07:01:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:17 dist-test-slave-sfvt krb5kdc[22242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:17 dist-test-slave-sfvt krb5kdc[22242](info): set up 2 sockets
Sep 26 07:01:17 dist-test-slave-sfvt krb5kdc[22242](info): commencing operation
krb5kdc: starting...
W20260926 07:01:18.599323 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:18 dist-test-slave-sfvt krb5kdc[22242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:18.718029 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:18.798972 22255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:18.799189 22255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:18.799212 22255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:18.800330 22255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:18.800365 22255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:18.800379 22255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:18.800391 22255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:18.800403 22255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:18.801772 22255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:18.801954 22255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:18.802194 22255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:18.806938 22260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:18.806906 22261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:18.806912 22263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:18.807111 22255 server_base.cc:1034] running on GCE node
I20260926 07:01:18.807435 22255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:18.807687 22255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:18.808856 22255 hybrid_clock.cc:648] HybridClock initialized: now 1790406078808842 us; error 42 us; skew 500 ppm
Sep 26 07:01:18 dist-test-slave-sfvt krb5kdc[22242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:18.811317 22255 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:18.811708 22255 webserver.cc:460] Webserver started at http://127.0.0.1:37389/ using document root <none> and password file <none>
I20260926 07:01:18.811893 22255 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:18.811934 22255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:18.812024 22255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:18.813036 22255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "961fe95ad68c435a86015fa123dbb06c"
format_stamp: "Formatted at 2026-09-26 07:01:18 on dist-test-slave-sfvt"
server_key: "43445692e3407a769ba8a8155d5152c0"
server_key_iv: "8e77756f1e4f7e8ac9d0648885129d67"
server_key_version: "encryptionkey@0"
I20260926 07:01:18.813345 22255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "961fe95ad68c435a86015fa123dbb06c"
format_stamp: "Formatted at 2026-09-26 07:01:18 on dist-test-slave-sfvt"
server_key: "43445692e3407a769ba8a8155d5152c0"
server_key_iv: "8e77756f1e4f7e8ac9d0648885129d67"
server_key_version: "encryptionkey@0"
I20260926 07:01:18.814625 22255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:18.815424 22270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:18.815600 22255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:18.815697 22255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "961fe95ad68c435a86015fa123dbb06c"
format_stamp: "Formatted at 2026-09-26 07:01:18 on dist-test-slave-sfvt"
server_key: "43445692e3407a769ba8a8155d5152c0"
server_key_iv: "8e77756f1e4f7e8ac9d0648885129d67"
server_key_version: "encryptionkey@0"
I20260926 07:01:18.815773 22255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:18.839859 22255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:18.841779 22255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:18.841930 22255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:18.845881 22255 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40527
I20260926 07:01:18.845944 22322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40527 every 8 connection(s)
I20260926 07:01:18.846302 22255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:18.847328 22323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:18.849547 22323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c: Bootstrap starting.
I20260926 07:01:18.850122 22323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:18.850410 22323 log.cc:824] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:18.851137 22323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c: No bootstrap required, opened a new log
I20260926 07:01:18.852891 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 22255
I20260926 07:01:18.853003 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:18.853195 1321 external_mini_cluster.cc:1428] Setting key 696e7cb8c96a505cb182823f777b78ea
I20260926 07:01:18.853170 22323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961fe95ad68c435a86015fa123dbb06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260926 07:01:18.853368 22323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:18.853416 22323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961fe95ad68c435a86015fa123dbb06c, State: Initialized, Role: FOLLOWER
I20260926 07:01:18.853559 22323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961fe95ad68c435a86015fa123dbb06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260926 07:01:18.853650 22323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:18.853682 22323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:18.853780 22323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:18.854831 22323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961fe95ad68c435a86015fa123dbb06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260926 07:01:18.854919 22323 leader_election.cc:302] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 961fe95ad68c435a86015fa123dbb06c; no voters:
I20260926 07:01:18.855093 22323 leader_election.cc:288] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:18.855134 22328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:18.855314 22328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [term 1 LEADER]: Becoming Leader. State: Replica: 961fe95ad68c435a86015fa123dbb06c, State: Running, Role: LEADER
I20260926 07:01:18.855365 22323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:18.855463 22328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961fe95ad68c435a86015fa123dbb06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260926 07:01:18.855958 22329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "961fe95ad68c435a86015fa123dbb06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961fe95ad68c435a86015fa123dbb06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } } }
I20260926 07:01:18.856122 22329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:18.856577 22337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:18.857093 22337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:18.857067 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 961fe95ad68c435a86015fa123dbb06c. Latest consensus state: current_term: 1 leader_uuid: "961fe95ad68c435a86015fa123dbb06c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961fe95ad68c435a86015fa123dbb06c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } } }
I20260926 07:01:18.857272 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:18.859023 22327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:18.853649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:39581 (local address 127.0.0.1:40527)
0926 07:01:18.853841 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:18.853851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:18.853932 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:01:18.854078 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:18.854084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:18.854135 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:18.854253 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:18.854526 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:18.855747 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:18.856865 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:18.857483 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:18.857618 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:18.858331 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:18.858334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:18.858335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:18.858559 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:18.858601 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:18.858618 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:18.858706 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:18.858786 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 07:01:18.858815 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260926 07:01:18.859045 22337 catalog_manager.cc:1347] Generated new cluster ID: 7c28ed15929648708e7dc74a96006609
I20260926 07:01:18.859097 22337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:18.860106 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:18.870226 22327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:18.863627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:37949 (local address 127.0.0.1:40527)
0926 07:01:18.863679 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:18.863681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:18.867473 (+ 3792us) server_negotiation.cc:407] Connection header received
0926 07:01:18.867542 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:18.867543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:18.867554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:18.867609 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:18.867763 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:18.868238 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:18.868949 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:18.869173 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:18.869252 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:18.869968 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:18.869969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:18.869970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:18.869975 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:18.869989 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:18.869994 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:18.870048 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:18.870107 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:01:18.870119 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:01:18.878139 22337 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:18.878826 22337 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:18.883379 22337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 961fe95ad68c435a86015fa123dbb06c: Generated new TSK 0
I20260926 07:01:18.883575 22337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:18.890784 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 22255
W20260926 07:01:18.895277 22240 connection.cc:535] client connection to 127.0.0.1:40527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40527): Connection reset by peer (error 104)
2026-09-26T07:01:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:18 dist-test-slave-sfvt krb5kdc[22354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:18 dist-test-slave-sfvt krb5kdc[22354](info): set up 2 sockets
Sep 26 07:01:18 dist-test-slave-sfvt krb5kdc[22354](info): commencing operation
krb5kdc: starting...
W20260926 07:01:19.931475 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:20 dist-test-slave-sfvt krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:20.046211 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:20.128209 22367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:20.128463 22367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:20.128504 22367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:20.129655 22367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:20.129729 22367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:20.129756 22367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:20.129787 22367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:20.129810 22367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:20.131378 22367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:20.131654 22367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:20.131881 22367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:20.137096 22375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:20.137089 22373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:20.137123 22367 server_base.cc:1034] running on GCE node
W20260926 07:01:20.137091 22372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:20.137531 22367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:20.137845 22367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:20.138986 22367 hybrid_clock.cc:648] HybridClock initialized: now 1790406080138971 us; error 38 us; skew 500 ppm
Sep 26 07:01:20 dist-test-slave-sfvt krb5kdc[22354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:20.141502 22367 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:20.141944 22367 webserver.cc:460] Webserver started at http://127.0.0.1:40991/ using document root <none> and password file <none>
I20260926 07:01:20.142191 22367 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:20.142237 22367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:20.142402 22367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:20.143563 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "4d302a9c176948538d36696f578c5148"
format_stamp: "Formatted at 2026-09-26 07:01:20 on dist-test-slave-sfvt"
server_key: "e173b362993e91fad3be85f8842df884"
server_key_iv: "ad84c6b37a057b9344afed041f4104f8"
server_key_version: "encryptionkey@0"
I20260926 07:01:20.143913 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "4d302a9c176948538d36696f578c5148"
format_stamp: "Formatted at 2026-09-26 07:01:20 on dist-test-slave-sfvt"
server_key: "e173b362993e91fad3be85f8842df884"
server_key_iv: "ad84c6b37a057b9344afed041f4104f8"
server_key_version: "encryptionkey@0"
I20260926 07:01:20.145260 22367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:20.146008 22382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:20.146219 22367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:20.146307 22367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "4d302a9c176948538d36696f578c5148"
format_stamp: "Formatted at 2026-09-26 07:01:20 on dist-test-slave-sfvt"
server_key: "e173b362993e91fad3be85f8842df884"
server_key_iv: "ad84c6b37a057b9344afed041f4104f8"
server_key_version: "encryptionkey@0"
I20260926 07:01:20.146381 22367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:20.149716 22367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:20.151748 22367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:20.151882 22367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:20.155681 22367 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39053
I20260926 07:01:20.155709 22434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39053 every 8 connection(s)
I20260926 07:01:20.156044 22367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:20.157059 22435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:20.159377 22435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148: Bootstrap starting.
I20260926 07:01:20.159956 22435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:20.160234 22435 log.cc:824] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:20.160669 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 22367
I20260926 07:01:20.160792 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:20.160989 1321 external_mini_cluster.cc:1428] Setting key cb599948b314bbd0f994afd2ae07d2ae
I20260926 07:01:20.161031 22435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148: No bootstrap required, opened a new log
I20260926 07:01:20.163236 22435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d302a9c176948538d36696f578c5148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260926 07:01:20.163465 22435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:20.163515 22435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d302a9c176948538d36696f578c5148, State: Initialized, Role: FOLLOWER
I20260926 07:01:20.163647 22435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d302a9c176948538d36696f578c5148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260926 07:01:20.163753 22435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:20.163784 22435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:20.163852 22435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:20.164170 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:20.161512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:41017 (local address 127.0.0.1:39053)
0926 07:01:20.161705 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:20.161715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:20.161751 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:01:20.162386 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:20.162396 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:20.162431 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:20.162589 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:20.163513 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:20.163517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:20.163518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:20.163522 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:20.163570 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:20.163584 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:20.163658 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:20.163725 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:01:20.163769 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260926 07:01:20.164604 22435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d302a9c176948538d36696f578c5148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260926 07:01:20.164726 22435 leader_election.cc:302] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d302a9c176948538d36696f578c5148; no voters:
I20260926 07:01:20.164925 22435 leader_election.cc:288] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:20.164950 22440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:20.165124 22440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [term 1 LEADER]: Becoming Leader. State: Replica: 4d302a9c176948538d36696f578c5148, State: Running, Role: LEADER
I20260926 07:01:20.165197 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:20.165253 22435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:20.165235 22440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d302a9c176948538d36696f578c5148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260926 07:01:20.165589 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d302a9c176948538d36696f578c5148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d302a9c176948538d36696f578c5148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } } }
I20260926 07:01:20.165663 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:20.165792 22440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d302a9c176948538d36696f578c5148. Latest consensus state: current_term: 1 leader_uuid: "4d302a9c176948538d36696f578c5148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d302a9c176948538d36696f578c5148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } } }
I20260926 07:01:20.165860 22440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148 [sys.catalog]: This master's current role is: LEADER
W20260926 07:01:20.167152 22460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:20.167244 22460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:20.167335 22446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:20.167536 22446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:20.167984 22439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:20.166836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:40783 (local address 127.0.0.1:39053)
0926 07:01:20.166878 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:20.166879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:20.167089 (+ 210us) server_negotiation.cc:407] Connection header received
0926 07:01:20.167121 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:20.167122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:20.167127 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:20.167190 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:20.167810 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:20.167811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:20.167811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:20.167814 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:20.167828 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:20.167832 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:20.167872 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:20.167933 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 07:01:20.167943 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:01:20.169070 22446 catalog_manager.cc:1347] Generated new cluster ID: 7a4a282c54f4450c91fb83d40f90f67c
I20260926 07:01:20.169119 22446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:20.187819 22446 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:20.188015 22446 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:20.196713 22446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d302a9c176948538d36696f578c5148: Generated new TSK 0
I20260926 07:01:20.196964 22446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:20.208125 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 22367
2026-09-26T07:01:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:20 dist-test-slave-sfvt krb5kdc[22466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:20 dist-test-slave-sfvt krb5kdc[22466](info): set up 2 sockets
Sep 26 07:01:20 dist-test-slave-sfvt krb5kdc[22466](info): commencing operation
krb5kdc: starting...
W20260926 07:01:21.255561 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:21 dist-test-slave-sfvt krb5kdc[22466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:21.379941 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:21.462672 22479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:21.462920 22479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:21.462946 22479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:21.464226 22479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:21.464274 22479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:21.464289 22479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:21.464303 22479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:21.464314 22479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:21.465919 22479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:21.466219 22479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:21.466485 22479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:21.471587 22484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:21.471771 22479 server_base.cc:1034] running on GCE node
W20260926 07:01:21.471594 22485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:21.471637 22487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:21.472187 22479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:21.472506 22479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:21.473681 22479 hybrid_clock.cc:648] HybridClock initialized: now 1790406081473675 us; error 38 us; skew 500 ppm
Sep 26 07:01:21 dist-test-slave-sfvt krb5kdc[22466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:21.479400 22479 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:21.479884 22479 webserver.cc:460] Webserver started at http://127.0.0.1:40515/ using document root <none> and password file <none>
I20260926 07:01:21.480080 22479 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:21.480120 22479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:21.480221 22479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:21.481251 22479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "e3c9a9e3b5a9450cad8a555468f28832"
format_stamp: "Formatted at 2026-09-26 07:01:21 on dist-test-slave-sfvt"
server_key: "ca1fdcc29242a163122fab4b162d53b0"
server_key_iv: "8ea00fc2c445a443a0c81112bd9c89e5"
server_key_version: "encryptionkey@0"
I20260926 07:01:21.481554 22479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "e3c9a9e3b5a9450cad8a555468f28832"
format_stamp: "Formatted at 2026-09-26 07:01:21 on dist-test-slave-sfvt"
server_key: "ca1fdcc29242a163122fab4b162d53b0"
server_key_iv: "8ea00fc2c445a443a0c81112bd9c89e5"
server_key_version: "encryptionkey@0"
I20260926 07:01:21.482884 22479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:21.483733 22494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:21.483911 22479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:01:21.484005 22479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "e3c9a9e3b5a9450cad8a555468f28832"
format_stamp: "Formatted at 2026-09-26 07:01:21 on dist-test-slave-sfvt"
server_key: "ca1fdcc29242a163122fab4b162d53b0"
server_key_iv: "8ea00fc2c445a443a0c81112bd9c89e5"
server_key_version: "encryptionkey@0"
I20260926 07:01:21.484064 22479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:21.510390 22479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:21.512810 22479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:21.512933 22479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:21.517022 22479 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35619
I20260926 07:01:21.517076 22546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35619 every 8 connection(s)
I20260926 07:01:21.517416 22479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:21.518467 22547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:21.520769 22547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832: Bootstrap starting.
I20260926 07:01:21.521359 22547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:21.521647 22547 log.cc:824] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:21.522347 22547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832: No bootstrap required, opened a new log
I20260926 07:01:21.524055 22547 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c9a9e3b5a9450cad8a555468f28832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35619 } }
I20260926 07:01:21.524219 22547 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:21.524266 22547 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c9a9e3b5a9450cad8a555468f28832, State: Initialized, Role: FOLLOWER
I20260926 07:01:21.524454 22547 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c9a9e3b5a9450cad8a555468f28832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35619 } }
I20260926 07:01:21.524556 22547 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:21.524601 22547 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:21.524652 22547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:21.525286 22547 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c9a9e3b5a9450cad8a555468f28832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35619 } }
I20260926 07:01:21.525341 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 22479
I20260926 07:01:21.525408 22547 leader_election.cc:302] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c9a9e3b5a9450cad8a555468f28832; no voters:
I20260926 07:01:21.525449 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:21.525583 22547 leader_election.cc:288] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:21.525636 1321 external_mini_cluster.cc:1428] Setting key e035f6e8b8688b49380581613c07799a
I20260926 07:01:21.525658 22550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:21.525863 22547 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:21.526372 22550 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [term 1 LEADER]: Becoming Leader. State: Replica: e3c9a9e3b5a9450cad8a555468f28832, State: Running, Role: LEADER
I20260926 07:01:21.526523 22550 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c9a9e3b5a9450cad8a555468f28832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35619 } }
I20260926 07:01:21.527100 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c9a9e3b5a9450cad8a555468f28832. Latest consensus state: current_term: 1 leader_uuid: "e3c9a9e3b5a9450cad8a555468f28832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c9a9e3b5a9450cad8a555468f28832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35619 } } }
I20260926 07:01:21.527180 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:21.527660 22551 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c9a9e3b5a9450cad8a555468f28832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c9a9e3b5a9450cad8a555468f28832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35619 } } }
I20260926 07:01:21.527976 22551 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832 [sys.catalog]: This master's current role is: LEADER
W20260926 07:01:21.528277 22568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:21.528335 22568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:21.528379 22562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:21.528523 22562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:21.531718 22562 catalog_manager.cc:1347] Generated new cluster ID: 097143a0417e4b07aaea0012d8aa79c0
I20260926 07:01:21.531777 22562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:21 dist-test-slave-sfvt krb5kdc[22466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:01:21.537025 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:21.526148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:49549 (local address 127.0.0.1:35619)
0926 07:01:21.526446 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:21.526454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:21.526484 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:01:21.526592 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:21.526618 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:21.526682 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:21.527805 (+ 1123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:21.527940 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:21.528992 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:21.529950 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:21.530362 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:21.530483 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:21.533069 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:21.533074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:21.533075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:21.533086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:21.535368 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:21.535778 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:21.535781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:21.535781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:21.535895 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:21.536135 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:21.536136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:21.536137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:21.536410 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:21.536589 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:21.536687 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 07:01:21.536798 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":95,"threads_started":1}
I20260926 07:01:21.538086 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:21.544193 22562 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:21.544891 22562 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:21.550153 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:21.539493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:36583 (local address 127.0.0.1:35619)
0926 07:01:21.539534 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:21.539536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:21.543467 (+ 3931us) server_negotiation.cc:407] Connection header received
0926 07:01:21.543548 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:21.543549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:21.543560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:21.543618 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:21.544573 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:21.545128 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:21.545843 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:21.546079 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:21.546166 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:21.547465 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:21.547467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:21.547467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:21.547474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:21.548940 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:21.549318 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:21.549320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:21.549320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:21.549388 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:21.549608 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:21.549609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:21.549609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:21.549722 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:21.549849 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:21.549973 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 07:01:21.550037 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:21.554315 22562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c9a9e3b5a9450cad8a555468f28832: Generated new TSK 0
I20260926 07:01:21.554512 22562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:21.561354 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 22479
W20260926 07:01:21.565575 22464 connection.cc:535] client connection to 127.0.0.1:35619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35619): Connection reset by peer (error 104)
2026-09-26T07:01:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:21 dist-test-slave-sfvt krb5kdc[22578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:21 dist-test-slave-sfvt krb5kdc[22578](info): set up 2 sockets
Sep 26 07:01:21 dist-test-slave-sfvt krb5kdc[22578](info): commencing operation
krb5kdc: starting...
W20260926 07:01:22.583237 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:22 dist-test-slave-sfvt krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:22.700219 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:22.778724 22591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:22.778920 22591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:22.778942 22591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:22.780138 22591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:22.780176 22591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:22.780191 22591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:22.780205 22591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:22.780216 22591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:22.781590 22591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:22.781769 22591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:22.781951 22591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:22.786772 22599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:22.786782 22597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:22.786790 22596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:22.787072 22591 server_base.cc:1034] running on GCE node
I20260926 07:01:22.787321 22591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:22.787570 22591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:22.788705 22591 hybrid_clock.cc:648] HybridClock initialized: now 1790406082788694 us; error 27 us; skew 500 ppm
Sep 26 07:01:22 dist-test-slave-sfvt krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:22.791059 22591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:22.791520 22591 webserver.cc:460] Webserver started at http://127.0.0.1:36347/ using document root <none> and password file <none>
I20260926 07:01:22.791736 22591 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:22.791783 22591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:22.791937 22591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:22.792932 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "57863ce52d8e4c1fba0edc31e925fb7f"
format_stamp: "Formatted at 2026-09-26 07:01:22 on dist-test-slave-sfvt"
server_key: "efbc852da9b9982c3e949b9469be93a5"
server_key_iv: "26faa5786f86d362c8910186592fb60e"
server_key_version: "encryptionkey@0"
I20260926 07:01:22.793288 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "57863ce52d8e4c1fba0edc31e925fb7f"
format_stamp: "Formatted at 2026-09-26 07:01:22 on dist-test-slave-sfvt"
server_key: "efbc852da9b9982c3e949b9469be93a5"
server_key_iv: "26faa5786f86d362c8910186592fb60e"
server_key_version: "encryptionkey@0"
I20260926 07:01:22.794601 22591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:22.795382 22606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:22.795571 22591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:22.795652 22591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "57863ce52d8e4c1fba0edc31e925fb7f"
format_stamp: "Formatted at 2026-09-26 07:01:22 on dist-test-slave-sfvt"
server_key: "efbc852da9b9982c3e949b9469be93a5"
server_key_iv: "26faa5786f86d362c8910186592fb60e"
server_key_version: "encryptionkey@0"
I20260926 07:01:22.795720 22591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:22.805259 22591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:22.807015 22591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:22.807120 22591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:22.810806 22591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42923
I20260926 07:01:22.810854 22658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42923 every 8 connection(s)
I20260926 07:01:22.811204 22591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:22.812140 22659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:22.814700 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 22591
I20260926 07:01:22.814795 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:22.814850 22659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f: Bootstrap starting.
I20260926 07:01:22.815017 1321 external_mini_cluster.cc:1428] Setting key c596af078393b20614beb1be4394b98f
I20260926 07:01:22.815634 22659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:22.816017 22659 log.cc:824] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:22.816766 22659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f: No bootstrap required, opened a new log
I20260926 07:01:22.819027 22659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57863ce52d8e4c1fba0edc31e925fb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42923 } }
I20260926 07:01:22.819192 22659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:22.819226 22659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57863ce52d8e4c1fba0edc31e925fb7f, State: Initialized, Role: FOLLOWER
I20260926 07:01:22.819375 22659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57863ce52d8e4c1fba0edc31e925fb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42923 } }
I20260926 07:01:22.819451 22659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:22.819480 22659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:22.819525 22659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:22.820267 22659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57863ce52d8e4c1fba0edc31e925fb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42923 } }
I20260926 07:01:22.820397 22659 leader_election.cc:302] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57863ce52d8e4c1fba0edc31e925fb7f; no voters:
I20260926 07:01:22.820566 22659 leader_election.cc:288] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:22.820618 22664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:22.820765 22664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [term 1 LEADER]: Becoming Leader. State: Replica: 57863ce52d8e4c1fba0edc31e925fb7f, State: Running, Role: LEADER
I20260926 07:01:22.820839 22659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:22.820916 22664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57863ce52d8e4c1fba0edc31e925fb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42923 } }
I20260926 07:01:22.821290 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:22.815561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:59313 (local address 127.0.0.1:42923)
0926 07:01:22.815763 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:22.815773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:22.815809 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:01:22.816399 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:22.816429 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:22.816496 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:22.816645 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:22.816822 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:22.818182 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:22.818933 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:22.819573 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:22.819683 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:22.820442 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:22.820446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:22.820446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:22.820655 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:22.820696 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:22.820712 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:22.820794 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:22.821040 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 07:01:22.821074 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260926 07:01:22.821390 22664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57863ce52d8e4c1fba0edc31e925fb7f. Latest consensus state: current_term: 1 leader_uuid: "57863ce52d8e4c1fba0edc31e925fb7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57863ce52d8e4c1fba0edc31e925fb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42923 } } }
I20260926 07:01:22.821870 22664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:22.821497 22666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57863ce52d8e4c1fba0edc31e925fb7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57863ce52d8e4c1fba0edc31e925fb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42923 } } }
I20260926 07:01:22.822163 22666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:22.822572 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 07:01:22.822660 22679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:22.822705 22679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:22.822794 22680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:22.822974 22680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:22.824488 22680 catalog_manager.cc:1347] Generated new cluster ID: c4b06052780945ff94fc5620ca721358
I20260926 07:01:22.824538 22680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:22.827239 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:22.824080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:51439 (local address 127.0.0.1:42923)
0926 07:01:22.824122 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:22.824124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:22.824526 (+ 402us) server_negotiation.cc:407] Connection header received
0926 07:01:22.824614 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:22.824615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:22.824625 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:22.824673 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:22.824849 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:22.825326 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:22.826033 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:22.826266 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:22.826339 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:22.826997 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:22.826998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:22.826999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:22.827003 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:22.827017 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:22.827021 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:22.827073 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:22.827136 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:01:22.827145 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:22.853148 22680 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:22.853894 22680 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:22.864980 22680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57863ce52d8e4c1fba0edc31e925fb7f: Generated new TSK 0
I20260926 07:01:22.865206 22680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:22.867524 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 22591
W20260926 07:01:22.871501 22576 connection.cc:535] client connection to 127.0.0.1:42923 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42923): Connection reset by peer (error 104)
2026-09-26T07:01:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:22 dist-test-slave-sfvt krb5kdc[22690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:22 dist-test-slave-sfvt krb5kdc[22690](info): set up 2 sockets
Sep 26 07:01:22 dist-test-slave-sfvt krb5kdc[22690](info): commencing operation
krb5kdc: starting...
W20260926 07:01:23.891453 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:23 dist-test-slave-sfvt krb5kdc[22690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:01:24.009584 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:24.092036 22703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:24.092252 22703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:24.092274 22703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:24.093627 22703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:24.093675 22703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:24.093690 22703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:24.093703 22703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:24.093715 22703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:24.095136 22703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:24.095417 22703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:24.095608 22703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:24.100350 22708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:24.100363 22709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:24.100610 22703 server_base.cc:1034] running on GCE node
W20260926 07:01:24.100371 22711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:24.100932 22703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:24.101257 22703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:24.102440 22703 hybrid_clock.cc:648] HybridClock initialized: now 1790406084102447 us; error 71 us; skew 500 ppm
Sep 26 07:01:24 dist-test-slave-sfvt krb5kdc[22690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:24.105036 22703 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:01:24.105482 22703 webserver.cc:460] Webserver started at http://127.0.0.1:42093/ using document root <none> and password file <none>
I20260926 07:01:24.105670 22703 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:24.105711 22703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:24.105798 22703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:24.106786 22703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "fcea16be491d4ea89350c733aca07001"
format_stamp: "Formatted at 2026-09-26 07:01:24 on dist-test-slave-sfvt"
server_key: "9a97224afbdbb9f084d73c0c6f7dc269"
server_key_iv: "a86cb20df318d5e1ac26225b7fbcdaa1"
server_key_version: "encryptionkey@0"
I20260926 07:01:24.107080 22703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "fcea16be491d4ea89350c733aca07001"
format_stamp: "Formatted at 2026-09-26 07:01:24 on dist-test-slave-sfvt"
server_key: "9a97224afbdbb9f084d73c0c6f7dc269"
server_key_iv: "a86cb20df318d5e1ac26225b7fbcdaa1"
server_key_version: "encryptionkey@0"
I20260926 07:01:24.108407 22703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:24.109104 22718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:24.109266 22703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:24.109370 22703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "fcea16be491d4ea89350c733aca07001"
format_stamp: "Formatted at 2026-09-26 07:01:24 on dist-test-slave-sfvt"
server_key: "9a97224afbdbb9f084d73c0c6f7dc269"
server_key_iv: "a86cb20df318d5e1ac26225b7fbcdaa1"
server_key_version: "encryptionkey@0"
I20260926 07:01:24.109443 22703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:24.112818 22703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:24.114792 22703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:24.114948 22703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:24.119014 22703 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45927
I20260926 07:01:24.119050 22770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45927 every 8 connection(s)
I20260926 07:01:24.119442 22703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:24.120461 22771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:24.122871 22771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001: Bootstrap starting.
I20260926 07:01:24.123548 22771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:24.123863 22771 log.cc:824] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:24.124395 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 22703
I20260926 07:01:24.124521 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:24.124687 22771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001: No bootstrap required, opened a new log
I20260926 07:01:24.124732 1321 external_mini_cluster.cc:1428] Setting key b0bd0860d1f193daaefd16264557e843
I20260926 07:01:24.127065 22771 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcea16be491d4ea89350c733aca07001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260926 07:01:24.127240 22771 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:24.127331 22771 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcea16be491d4ea89350c733aca07001, State: Initialized, Role: FOLLOWER
I20260926 07:01:24.127524 22771 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcea16be491d4ea89350c733aca07001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260926 07:01:24.127604 22771 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:24.127625 22771 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:24.127691 22771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:24.127775 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:24.125256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:56903 (local address 127.0.0.1:45927)
0926 07:01:24.125471 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:24.125480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:24.125513 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:01:24.126079 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:24.126086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:24.126110 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:24.126266 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:24.126931 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:24.126941 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:24.126942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:24.126945 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:24.126979 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:24.126990 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:24.127066 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:24.127135 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 07:01:24.127178 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
I20260926 07:01:24.128449 22771 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcea16be491d4ea89350c733aca07001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260926 07:01:24.128554 22771 leader_election.cc:302] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcea16be491d4ea89350c733aca07001; no voters:
I20260926 07:01:24.128728 22771 leader_election.cc:288] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:24.128738 22776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:24.128845 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:24.128893 22776 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [term 1 LEADER]: Becoming Leader. State: Replica: fcea16be491d4ea89350c733aca07001, State: Running, Role: LEADER
I20260926 07:01:24.128964 22776 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcea16be491d4ea89350c733aca07001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260926 07:01:24.128993 22771 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:24.129267 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcea16be491d4ea89350c733aca07001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcea16be491d4ea89350c733aca07001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260926 07:01:24.129336 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:24.129487 22778 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcea16be491d4ea89350c733aca07001. Latest consensus state: current_term: 1 leader_uuid: "fcea16be491d4ea89350c733aca07001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcea16be491d4ea89350c733aca07001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260926 07:01:24.129536 22778 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:24.129525 22780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:24.130056 22780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:24.131757 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:24.130488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:33679 (local address 127.0.0.1:45927)
0926 07:01:24.130524 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:24.130526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:24.130725 (+ 199us) server_negotiation.cc:407] Connection header received
0926 07:01:24.130775 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:24.130775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:24.130780 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:24.130830 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:24.131517 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:24.131518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:24.131519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:24.131523 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:24.131544 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:24.131549 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:24.131608 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:24.131676 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 07:01:24.131689 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:24.131959 22780 catalog_manager.cc:1347] Generated new cluster ID: 7ace2bf7970147e2af3d74693ec72f11
I20260926 07:01:24.132023 22780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:24.140512 22780 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:24.140683 22780 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:24.143723 22780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcea16be491d4ea89350c733aca07001: Generated new TSK 0
I20260926 07:01:24.143901 22780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:24.151584 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 22703
2026-09-26T07:01:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:24 dist-test-slave-sfvt krb5kdc[22802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:24 dist-test-slave-sfvt krb5kdc[22802](info): set up 2 sockets
Sep 26 07:01:24 dist-test-slave-sfvt krb5kdc[22802](info): commencing operation
krb5kdc: starting...
W20260926 07:01:25.187474 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:25 dist-test-slave-sfvt krb5kdc[22802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:25.306926 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36257
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:36257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:25.386757 22815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:25.387015 22815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:25.387054 22815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:25.388240 22815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:25.388316 22815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:25.388340 22815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:25.388362 22815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:25.388396 22815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:25.389853 22815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:36257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:25.390130 22815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:25.390355 22815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:25.395026 22823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:25.395079 22821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:25.395072 22820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:25.395351 22815 server_base.cc:1034] running on GCE node
I20260926 07:01:25.395597 22815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:25.395891 22815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:25.397066 22815 hybrid_clock.cc:648] HybridClock initialized: now 1790406085397051 us; error 44 us; skew 500 ppm
Sep 26 07:01:25 dist-test-slave-sfvt krb5kdc[22802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:25.399880 22815 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:25.400368 22815 webserver.cc:460] Webserver started at http://127.1.74.126:42499/ using document root <none> and password file <none>
I20260926 07:01:25.400609 22815 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:25.400682 22815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:25.400839 22815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:25.401918 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "7551756095564cf782ba55561eb84889"
format_stamp: "Formatted at 2026-09-26 07:01:25 on dist-test-slave-sfvt"
server_key: "dc8869ee581d89282924d54b70f9f2a8"
server_key_iv: "a02388adec26259371b41e301ad4bcc7"
server_key_version: "encryptionkey@0"
I20260926 07:01:25.402289 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "7551756095564cf782ba55561eb84889"
format_stamp: "Formatted at 2026-09-26 07:01:25 on dist-test-slave-sfvt"
server_key: "dc8869ee581d89282924d54b70f9f2a8"
server_key_iv: "a02388adec26259371b41e301ad4bcc7"
server_key_version: "encryptionkey@0"
I20260926 07:01:25.403690 22815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:25.404399 22830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:25.404558 22815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:25.404657 22815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "7551756095564cf782ba55561eb84889"
format_stamp: "Formatted at 2026-09-26 07:01:25 on dist-test-slave-sfvt"
server_key: "dc8869ee581d89282924d54b70f9f2a8"
server_key_iv: "a02388adec26259371b41e301ad4bcc7"
server_key_version: "encryptionkey@0"
I20260926 07:01:25.404738 22815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:25.413743 22815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:25.415645 22815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:25.415814 22815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:25.419793 22815 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:36257
I20260926 07:01:25.419823 22882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:36257 every 8 connection(s)
I20260926 07:01:25.420176 22815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:25.421005 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:25.421451 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 22815
I20260926 07:01:25.421581 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:25.421759 1321 external_mini_cluster.cc:1428] Setting key f6a243c47237a302030eff615ad3d882
I20260926 07:01:25.423648 22883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889: Bootstrap starting.
I20260926 07:01:25.424214 22883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:25.424505 22883 log.cc:824] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:25.425194 22883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889: No bootstrap required, opened a new log
I20260926 07:01:25.427142 22883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7551756095564cf782ba55561eb84889" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36257 } }
I20260926 07:01:25.427376 22883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:25.427465 22883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7551756095564cf782ba55561eb84889, State: Initialized, Role: FOLLOWER
I20260926 07:01:25.427615 22883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7551756095564cf782ba55561eb84889" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36257 } }
I20260926 07:01:25.427723 22883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:25.427767 22883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:25.427815 22883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 07:01:25 dist-test-slave-sfvt krb5kdc[22802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:25.428370 22883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7551756095564cf782ba55561eb84889" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36257 } }
I20260926 07:01:25.428470 22883 leader_election.cc:302] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7551756095564cf782ba55561eb84889; no voters:
I20260926 07:01:25.428663 22883 leader_election.cc:288] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:25.428691 22888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:25.428802 22888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [term 1 LEADER]: Becoming Leader. State: Replica: 7551756095564cf782ba55561eb84889, State: Running, Role: LEADER
I20260926 07:01:25.428930 22883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:25.428946 22888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7551756095564cf782ba55561eb84889" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36257 } }
I20260926 07:01:25.429692 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7551756095564cf782ba55561eb84889. Latest consensus state: current_term: 1 leader_uuid: "7551756095564cf782ba55561eb84889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7551756095564cf782ba55561eb84889" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36257 } } }
I20260926 07:01:25.429780 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:25.430294 22897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:25.430294 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7551756095564cf782ba55561eb84889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7551756095564cf782ba55561eb84889" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 36257 } } }
I20260926 07:01:25.430451 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:25.431003 22897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:25.432693 22897 catalog_manager.cc:1347] Generated new cluster ID: b9b4d359e921494a99866cf8fa6aa77b
I20260926 07:01:25.432747 22897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:25.433254 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:25.422275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59567 (local address 127.1.74.126:36257)
0926 07:01:25.422461 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:25.422477 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:25.422502 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:01:25.422986 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:25.423000 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:25.423046 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:25.423191 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:25.423865 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:25.424848 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:25.425692 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:25.426118 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:25.426233 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:25.428930 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:25.428934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:25.428934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:25.428947 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:25.431563 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:25.431991 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:25.431994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:25.431995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:25.432103 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:25.432371 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:25.432372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:25.432373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:25.432667 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:25.432813 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:25.432980 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 07:01:25.433089 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260926 07:01:25.434162 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:25.441648 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:25.435515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59519 (local address 127.1.74.126:36257)
0926 07:01:25.435566 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:25.435568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:25.435690 (+ 122us) server_negotiation.cc:407] Connection header received
0926 07:01:25.435762 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:25.435762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:25.435771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:25.435816 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:25.436325 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:25.436805 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:25.437527 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:25.437769 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:25.437844 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:25.439019 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:25.439020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:25.439021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:25.439028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:25.440481 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:25.440846 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:25.440847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:25.440847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:25.440908 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:25.441126 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:25.441127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:25.441127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:25.441240 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:25.441349 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:25.441470 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 07:01:25.441532 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 07:01:25.463290 22897 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:25.463845 22897 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:25.472877 22897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7551756095564cf782ba55561eb84889: Generated new TSK 0
I20260926 07:01:25.473099 22897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:25.475967 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 22815
W20260926 07:01:25.479806 22800 connection.cc:535] client connection to 127.1.74.126:36257 recv error: Network error: failed to read from TLS socket (remote: 127.1.74.126:36257): Connection reset by peer (error 104)
2026-09-26T07:01:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:25 dist-test-slave-sfvt krb5kdc[22914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:25 dist-test-slave-sfvt krb5kdc[22914](info): set up 2 sockets
Sep 26 07:01:25 dist-test-slave-sfvt krb5kdc[22914](info): commencing operation
krb5kdc: starting...
W20260926 07:01:26.507591 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:26 dist-test-slave-sfvt krb5kdc[22914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:26.626281 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46845
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:26.705454 22927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:26.705701 22927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:26.705739 22927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:26.706918 22927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:26.706990 22927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:26.707013 22927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:26.707034 22927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:26.707064 22927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:26.708564 22927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:46845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:26.708838 22927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:26.709065 22927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:26.713855 22933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:26.713907 22932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:26.713855 22935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:26.714105 22927 server_base.cc:1034] running on GCE node
I20260926 07:01:26.714362 22927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:26.714668 22927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:26.715831 22927 hybrid_clock.cc:648] HybridClock initialized: now 1790406086715818 us; error 26 us; skew 500 ppm
Sep 26 07:01:26 dist-test-slave-sfvt krb5kdc[22914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:26.722731 22927 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:26.723230 22927 webserver.cc:460] Webserver started at http://127.1.74.126:45741/ using document root <none> and password file <none>
I20260926 07:01:26.723505 22927 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:26.723572 22927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:26.723728 22927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:26.724812 22927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "b73b6b88482843d4b9567f2c299b56df"
format_stamp: "Formatted at 2026-09-26 07:01:26 on dist-test-slave-sfvt"
server_key: "6e3dc843c5b5733ead9760c23dfe2e48"
server_key_iv: "9e1ed3fb7bac38fbd654ecb73d01fb73"
server_key_version: "encryptionkey@0"
I20260926 07:01:26.725207 22927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "b73b6b88482843d4b9567f2c299b56df"
format_stamp: "Formatted at 2026-09-26 07:01:26 on dist-test-slave-sfvt"
server_key: "6e3dc843c5b5733ead9760c23dfe2e48"
server_key_iv: "9e1ed3fb7bac38fbd654ecb73d01fb73"
server_key_version: "encryptionkey@0"
I20260926 07:01:26.726500 22927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:26.727227 22942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:26.727424 22927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:26.727520 22927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "b73b6b88482843d4b9567f2c299b56df"
format_stamp: "Formatted at 2026-09-26 07:01:26 on dist-test-slave-sfvt"
server_key: "6e3dc843c5b5733ead9760c23dfe2e48"
server_key_iv: "9e1ed3fb7bac38fbd654ecb73d01fb73"
server_key_version: "encryptionkey@0"
I20260926 07:01:26.727600 22927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:26.763332 22927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:26.765208 22927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:26.765379 22927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:26.769520 22927 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:46845
I20260926 07:01:26.769559 22994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:46845 every 8 connection(s)
I20260926 07:01:26.769956 22927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:26.770784 22995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:26.771442 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 22927
I20260926 07:01:26.771556 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:26.771733 1321 external_mini_cluster.cc:1428] Setting key 4417e269ef9f591487bd4ae817d40462
I20260926 07:01:26.773666 22995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df: Bootstrap starting.
I20260926 07:01:26.774328 22995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:26.774624 22995 log.cc:824] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:26.775348 22995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df: No bootstrap required, opened a new log
I20260926 07:01:26.777118 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:26.772251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54749 (local address 127.1.74.126:46845)
0926 07:01:26.772420 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:26.772428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:26.772452 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:01:26.772841 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:26.772847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:26.772892 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:26.773003 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:26.773253 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:26.774174 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:26.775179 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:26.775676 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:26.775763 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:26.776488 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:26.776492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:26.776492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:26.776644 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:26.776672 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:26.776682 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:26.776755 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:26.776871 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 07:01:26.776915 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260926 07:01:26.777743 22995 raft_consensus.cc:348] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73b6b88482843d4b9567f2c299b56df" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46845 } }
I20260926 07:01:26.777948 22995 raft_consensus.cc:374] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:26.777995 22995 raft_consensus.cc:729] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73b6b88482843d4b9567f2c299b56df, State: Initialized, Role: FOLLOWER
I20260926 07:01:26.778120 22995 consensus_queue.cc:260] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73b6b88482843d4b9567f2c299b56df" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46845 } }
I20260926 07:01:26.778201 22995 raft_consensus.cc:388] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:26.778195 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:26.778242 22995 raft_consensus.cc:482] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:26.778296 22995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:26.778800 22995 raft_consensus.cc:504] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73b6b88482843d4b9567f2c299b56df" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46845 } }
I20260926 07:01:26.778913 22995 leader_election.cc:302] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b73b6b88482843d4b9567f2c299b56df; no voters:
I20260926 07:01:26.779084 22995 leader_election.cc:288] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:26.779127 23000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:26.779325 23000 raft_consensus.cc:686] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [term 1 LEADER]: Becoming Leader. State: Replica: b73b6b88482843d4b9567f2c299b56df, State: Running, Role: LEADER
I20260926 07:01:26.779446 23000 consensus_queue.cc:237] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73b6b88482843d4b9567f2c299b56df" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46845 } }
I20260926 07:01:26.779445 22995 sys_catalog.cc:564] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:26.781311 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [sys.catalog]: SysCatalogTable state changed. Reason: New leader b73b6b88482843d4b9567f2c299b56df. Latest consensus state: current_term: 1 leader_uuid: "b73b6b88482843d4b9567f2c299b56df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73b6b88482843d4b9567f2c299b56df" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46845 } } }
I20260926 07:01:26.781447 23004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b73b6b88482843d4b9567f2c299b56df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73b6b88482843d4b9567f2c299b56df" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46845 } } }
I20260926 07:01:26.781477 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:26.781491 23004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df [sys.catalog]: This master's current role is: LEADER
W20260926 07:01:26.782248 23019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:26.782308 23019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:26.782721 23020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:26.782968 23020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:26.783061 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:26.779704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41983 (local address 127.1.74.126:46845)
0926 07:01:26.779750 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:26.779752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:26.779786 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:01:26.779852 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:26.779852 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:26.779861 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:26.779928 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:26.780036 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:26.780789 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:26.781777 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:26.782042 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:26.782134 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:26.782797 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:26.782798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:26.782799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:26.782803 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:26.782816 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:26.782820 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:26.782872 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:26.782932 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:01:26.782941 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 07:01:26.784667 23020 catalog_manager.cc:1347] Generated new cluster ID: 5dcdd79bdc7b4a9c9d7b1dd229c04b14
I20260926 07:01:26.784719 23020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:26.804795 23020 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:26.805351 23020 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:26.816105 23020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b73b6b88482843d4b9567f2c299b56df: Generated new TSK 0
I20260926 07:01:26.816339 23020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:26.821803 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 22927
W20260926 07:01:26.826225 22912 connection.cc:535] client connection to 127.1.74.126:46845 recv error: Network error: failed to read from TLS socket (remote: 127.1.74.126:46845): Connection reset by peer (error 104)
2026-09-26T07:01:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:26 dist-test-slave-sfvt krb5kdc[23026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:26 dist-test-slave-sfvt krb5kdc[23026](info): set up 2 sockets
Sep 26 07:01:26 dist-test-slave-sfvt krb5kdc[23026](info): commencing operation
krb5kdc: starting...
W20260926 07:01:27.851322 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:27 dist-test-slave-sfvt krb5kdc[23026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:27.969854 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:39253
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:39253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:28.050179 23039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:28.050439 23039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:28.050480 23039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:28.051705 23039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:28.051774 23039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:28.051797 23039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:28.051817 23039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:28.051846 23039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:28.053288 23039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:39253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:39253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:28.053573 23039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:28.053797 23039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:28.058634 23045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:28.058679 23044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:28.058782 23039 server_base.cc:1034] running on GCE node
W20260926 07:01:28.058712 23047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:28.059151 23039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:28.059463 23039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:28.060657 23039 hybrid_clock.cc:648] HybridClock initialized: now 1790406088060631 us; error 40 us; skew 500 ppm
Sep 26 07:01:28 dist-test-slave-sfvt krb5kdc[23026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406088, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:28.063102 23039 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:28.063560 23039 webserver.cc:460] Webserver started at http://127.1.74.126:44433/ using document root <none> and password file <none>
I20260926 07:01:28.063750 23039 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:28.063791 23039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:28.063951 23039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:28.064927 23039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "e1d4252163c046118ee11c630dcff3fe"
format_stamp: "Formatted at 2026-09-26 07:01:28 on dist-test-slave-sfvt"
server_key: "8c2bbad7bd0a95bbdef8c0f7aa873b0d"
server_key_iv: "1cff582ba52ad3a5074130694713c665"
server_key_version: "encryptionkey@0"
I20260926 07:01:28.065225 23039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "e1d4252163c046118ee11c630dcff3fe"
format_stamp: "Formatted at 2026-09-26 07:01:28 on dist-test-slave-sfvt"
server_key: "8c2bbad7bd0a95bbdef8c0f7aa873b0d"
server_key_iv: "1cff582ba52ad3a5074130694713c665"
server_key_version: "encryptionkey@0"
I20260926 07:01:28.066567 23039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:28.067330 23054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:28.067539 23039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:28.067637 23039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "e1d4252163c046118ee11c630dcff3fe"
format_stamp: "Formatted at 2026-09-26 07:01:28 on dist-test-slave-sfvt"
server_key: "8c2bbad7bd0a95bbdef8c0f7aa873b0d"
server_key_iv: "1cff582ba52ad3a5074130694713c665"
server_key_version: "encryptionkey@0"
I20260926 07:01:28.067703 23039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:28.070935 23039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:28.072803 23039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:28.072913 23039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:28.076583 23039 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:39253
I20260926 07:01:28.076678 23106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:39253 every 8 connection(s)
I20260926 07:01:28.076956 23039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:28.077901 23107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:28.080134 23107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe: Bootstrap starting.
I20260926 07:01:28.080681 23107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:28.080958 23107 log.cc:824] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:28.081605 23107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe: No bootstrap required, opened a new log
I20260926 07:01:28.083175 23107 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d4252163c046118ee11c630dcff3fe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 39253 } }
I20260926 07:01:28.083388 23107 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:28.083441 23107 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d4252163c046118ee11c630dcff3fe, State: Initialized, Role: FOLLOWER
I20260926 07:01:28.083683 23107 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d4252163c046118ee11c630dcff3fe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 39253 } }
I20260926 07:01:28.083814 23107 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:28.083860 23107 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:28.083920 23107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:28.084225 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 23039
I20260926 07:01:28.084340 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:28.084537 1321 external_mini_cluster.cc:1428] Setting key a60190fd9720bf91f4d2eadd80ad1127
I20260926 07:01:28.085176 23107 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d4252163c046118ee11c630dcff3fe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 39253 } }
I20260926 07:01:28.085310 23107 leader_election.cc:302] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1d4252163c046118ee11c630dcff3fe; no voters:
I20260926 07:01:28.085484 23107 leader_election.cc:288] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:28.085505 23112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:28.085640 23112 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [term 1 LEADER]: Becoming Leader. State: Replica: e1d4252163c046118ee11c630dcff3fe, State: Running, Role: LEADER
I20260926 07:01:28.085762 23107 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:28.085711 23112 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d4252163c046118ee11c630dcff3fe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 39253 } }
I20260926 07:01:28.086498 23113 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1d4252163c046118ee11c630dcff3fe. Latest consensus state: current_term: 1 leader_uuid: "e1d4252163c046118ee11c630dcff3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d4252163c046118ee11c630dcff3fe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 39253 } } }
I20260926 07:01:28.086594 23113 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:28.086854 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:28.085027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53881 (local address 127.1.74.126:39253)
0926 07:01:28.085213 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:28.085220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:28.085264 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:01:28.085400 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:28.085407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:28.085436 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:28.085587 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:28.086316 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:28.086320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:28.086320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:28.086325 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:28.086374 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:28.086391 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:28.086451 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:28.086625 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 07:01:28.086666 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260926 07:01:28.087057 23121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:28.087493 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1d4252163c046118ee11c630dcff3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d4252163c046118ee11c630dcff3fe" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 39253 } } }
I20260926 07:01:28.087639 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:28.087848 23121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:28.088028 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:28.089645 23121 catalog_manager.cc:1347] Generated new cluster ID: cf6749bae4d94eb48091d005a43d72b0
I20260926 07:01:28.089694 23121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:28.090615 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:28.089493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45681 (local address 127.1.74.126:39253)
0926 07:01:28.089536 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:28.089539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:28.089708 (+ 169us) server_negotiation.cc:407] Connection header received
0926 07:01:28.089782 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:28.089782 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:28.089801 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:28.089855 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:28.090400 (+ 545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:28.090401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:28.090402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:28.090406 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:28.090418 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:28.090422 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:28.090477 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:28.090533 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 07:01:28.090547 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:28.109714 23121 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:28.109899 23121 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:28.120416 23121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1d4252163c046118ee11c630dcff3fe: Generated new TSK 0
I20260926 07:01:28.120622 23121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:28.125382 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 23039
2026-09-26T07:01:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:28 dist-test-slave-sfvt krb5kdc[23138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:28 dist-test-slave-sfvt krb5kdc[23138](info): set up 2 sockets
Sep 26 07:01:28 dist-test-slave-sfvt krb5kdc[23138](info): commencing operation
krb5kdc: starting...
W20260926 07:01:29.147538 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:29 dist-test-slave-sfvt krb5kdc[23138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:29.266623 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:44863
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:44863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:29.348728 23151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:29.348941 23151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:29.348964 23151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:29.350133 23151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:29.350176 23151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:29.350190 23151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:29.350203 23151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:29.350255 23151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:29.351845 23151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:44863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:29.352098 23151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:29.352329 23151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:29.357107 23157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:29.357131 23156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:29.357239 23151 server_base.cc:1034] running on GCE node
W20260926 07:01:29.357141 23159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:29.357534 23151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:29.357826 23151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:29.358989 23151 hybrid_clock.cc:648] HybridClock initialized: now 1790406089358986 us; error 39 us; skew 500 ppm
Sep 26 07:01:29 dist-test-slave-sfvt krb5kdc[23138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:29.361408 23151 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:29.361814 23151 webserver.cc:460] Webserver started at http://127.1.74.126:45659/ using document root <none> and password file <none>
I20260926 07:01:29.362030 23151 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:29.362097 23151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:29.362246 23151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:29.363390 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "103b6377a89a490ba410804f4bf059b3"
format_stamp: "Formatted at 2026-09-26 07:01:29 on dist-test-slave-sfvt"
server_key: "b975ba2316cbd905ab206f1b76fde4d2"
server_key_iv: "659edb511ac9389191b50eaa8f26a8a7"
server_key_version: "encryptionkey@0"
I20260926 07:01:29.363744 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "103b6377a89a490ba410804f4bf059b3"
format_stamp: "Formatted at 2026-09-26 07:01:29 on dist-test-slave-sfvt"
server_key: "b975ba2316cbd905ab206f1b76fde4d2"
server_key_iv: "659edb511ac9389191b50eaa8f26a8a7"
server_key_version: "encryptionkey@0"
I20260926 07:01:29.365042 23151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:29.365764 23166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:29.365933 23151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:29.366035 23151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "103b6377a89a490ba410804f4bf059b3"
format_stamp: "Formatted at 2026-09-26 07:01:29 on dist-test-slave-sfvt"
server_key: "b975ba2316cbd905ab206f1b76fde4d2"
server_key_iv: "659edb511ac9389191b50eaa8f26a8a7"
server_key_version: "encryptionkey@0"
I20260926 07:01:29.366107 23151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:29.385126 23151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:29.386900 23151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:29.387063 23151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:29.390765 23151 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:44863
I20260926 07:01:29.390800 23218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:44863 every 8 connection(s)
I20260926 07:01:29.391137 23151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:29.392055 23219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:29.394290 23219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3: Bootstrap starting.
I20260926 07:01:29.394852 23219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:29.395141 23219 log.cc:824] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:29.395833 23219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3: No bootstrap required, opened a new log
I20260926 07:01:29.397353 23219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103b6377a89a490ba410804f4bf059b3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44863 } }
I20260926 07:01:29.397485 23219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:29.397533 23219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103b6377a89a490ba410804f4bf059b3, State: Initialized, Role: FOLLOWER
I20260926 07:01:29.397684 23219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103b6377a89a490ba410804f4bf059b3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44863 } }
I20260926 07:01:29.397765 23219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:29.397816 23219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:29.397864 23219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:29.398656 23219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103b6377a89a490ba410804f4bf059b3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44863 } }
I20260926 07:01:29.398767 23219 leader_election.cc:302] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 103b6377a89a490ba410804f4bf059b3; no voters:
I20260926 07:01:29.398960 23219 leader_election.cc:288] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:29.398993 23222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:29.399143 23222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [term 1 LEADER]: Becoming Leader. State: Replica: 103b6377a89a490ba410804f4bf059b3, State: Running, Role: LEADER
I20260926 07:01:29.399219 23219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:29.399286 23222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103b6377a89a490ba410804f4bf059b3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44863 } }
I20260926 07:01:29.399654 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103b6377a89a490ba410804f4bf059b3. Latest consensus state: current_term: 1 leader_uuid: "103b6377a89a490ba410804f4bf059b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103b6377a89a490ba410804f4bf059b3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44863 } } }
I20260926 07:01:29.399657 23223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "103b6377a89a490ba410804f4bf059b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103b6377a89a490ba410804f4bf059b3" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44863 } } }
I20260926 07:01:29.399762 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:29.399770 23223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:29.400221 23229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:29.400764 23229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:29.401366 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 23151
I20260926 07:01:29.401479 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:29.401631 1321 external_mini_cluster.cc:1428] Setting key 935f90093ce1f32f810a45315cd7cef8
I20260926 07:01:29.402729 23229 catalog_manager.cc:1347] Generated new cluster ID: 9bf83e3c735e43878cc3ca00f383bcbc
I20260926 07:01:29.402781 23229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:29 dist-test-slave-sfvt krb5kdc[23138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:29.414067 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:29.402082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43629 (local address 127.1.74.126:44863)
0926 07:01:29.402326 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:29.402334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:29.402394 (+ 60us) server_negotiation.cc:407] Connection header received
0926 07:01:29.402517 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:29.402524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:29.402578 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:29.404146 (+ 1568us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:29.404243 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:29.405789 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:29.406925 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:29.407542 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:29.407980 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:29.410482 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:29.410486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:29.410487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:29.410498 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:29.412458 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:29.412831 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:29.412835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:29.412835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:29.412949 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:29.413147 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:29.413148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:29.413148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:29.413441 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:29.413581 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:29.413688 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 07:01:29.413822 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260926 07:01:29.415199 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:29.423331 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:29.416612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51145 (local address 127.1.74.126:44863)
0926 07:01:29.416645 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:29.416646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:29.416774 (+ 128us) server_negotiation.cc:407] Connection header received
0926 07:01:29.416799 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:29.416800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:29.416810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:29.416884 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:29.417427 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:29.418169 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:29.419105 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:29.419453 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:29.419514 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:29.420729 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:29.420731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:29.420731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:29.420738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:29.422242 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:29.422602 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:29.422603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:29.422603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:29.422656 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:29.422839 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:29.422840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:29.422840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:29.422958 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:29.423052 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:29.423156 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 07:01:29.423225 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:29.439637 23229 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:29.440351 23229 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:29.460378 23229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 103b6377a89a490ba410804f4bf059b3: Generated new TSK 0
I20260926 07:01:29.460614 23229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:29.467698 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 23151
2026-09-26T07:01:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:29 dist-test-slave-sfvt krb5kdc[23250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:29 dist-test-slave-sfvt krb5kdc[23250](info): set up 2 sockets
Sep 26 07:01:29 dist-test-slave-sfvt krb5kdc[23250](info): commencing operation
krb5kdc: starting...
W20260926 07:01:30.499332 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:30 dist-test-slave-sfvt krb5kdc[23250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:30.618480 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:43927
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:43927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:30.697789 23263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:30.697999 23263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:30.698022 23263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:30.699105 23263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:30.699139 23263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:30.699153 23263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:30.699167 23263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:30.699178 23263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:30.700585 23263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:43927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:43927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:30.700786 23263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:30.700954 23263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:30.705735 23269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:30.705726 23268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:30.705924 23263 server_base.cc:1034] running on GCE node
W20260926 07:01:30.705726 23271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:30.706228 23263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:30.706530 23263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:30.707777 23263 hybrid_clock.cc:648] HybridClock initialized: now 1790406090707765 us; error 36 us; skew 500 ppm
Sep 26 07:01:30 dist-test-slave-sfvt krb5kdc[23250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:30.710433 23263 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:30.710868 23263 webserver.cc:460] Webserver started at http://127.1.74.126:37815/ using document root <none> and password file <none>
I20260926 07:01:30.711113 23263 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:30.711225 23263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:30.711439 23263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:30.712563 23263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "eac006ac64c34290ab947dc9784f353b"
format_stamp: "Formatted at 2026-09-26 07:01:30 on dist-test-slave-sfvt"
server_key: "5ca5bbaf059b089ab71df57e22c4065f"
server_key_iv: "06e8b2726bf4c51448cd3abec971cb66"
server_key_version: "encryptionkey@0"
I20260926 07:01:30.712986 23263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "eac006ac64c34290ab947dc9784f353b"
format_stamp: "Formatted at 2026-09-26 07:01:30 on dist-test-slave-sfvt"
server_key: "5ca5bbaf059b089ab71df57e22c4065f"
server_key_iv: "06e8b2726bf4c51448cd3abec971cb66"
server_key_version: "encryptionkey@0"
I20260926 07:01:30.714356 23263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:30.715137 23278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:30.715351 23263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:30.715456 23263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "eac006ac64c34290ab947dc9784f353b"
format_stamp: "Formatted at 2026-09-26 07:01:30 on dist-test-slave-sfvt"
server_key: "5ca5bbaf059b089ab71df57e22c4065f"
server_key_iv: "06e8b2726bf4c51448cd3abec971cb66"
server_key_version: "encryptionkey@0"
I20260926 07:01:30.715538 23263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:30.726200 23263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:30.728255 23263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:30.728552 23263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:30.733045 23263 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:43927
I20260926 07:01:30.733096 23330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:43927 every 8 connection(s)
I20260926 07:01:30.733421 23263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:30.734284 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:30.736606 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b: Bootstrap starting.
I20260926 07:01:30.737177 23331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:30.737470 23331 log.cc:824] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:30.738166 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b: No bootstrap required, opened a new log
I20260926 07:01:30.739807 23331 raft_consensus.cc:348] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac006ac64c34290ab947dc9784f353b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 43927 } }
I20260926 07:01:30.739964 23331 raft_consensus.cc:374] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:30.740010 23331 raft_consensus.cc:729] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac006ac64c34290ab947dc9784f353b, State: Initialized, Role: FOLLOWER
I20260926 07:01:30.740176 23331 consensus_queue.cc:260] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac006ac64c34290ab947dc9784f353b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 43927 } }
I20260926 07:01:30.740262 23331 raft_consensus.cc:388] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:30.740299 23331 raft_consensus.cc:482] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:30.740351 23331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:30.741168 23331 raft_consensus.cc:504] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac006ac64c34290ab947dc9784f353b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 43927 } }
I20260926 07:01:30.741287 23331 leader_election.cc:302] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eac006ac64c34290ab947dc9784f353b; no voters:
I20260926 07:01:30.741467 23331 leader_election.cc:288] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:30.741501 23334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:30.741670 23334 raft_consensus.cc:686] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [term 1 LEADER]: Becoming Leader. State: Replica: eac006ac64c34290ab947dc9784f353b, State: Running, Role: LEADER
I20260926 07:01:30.741730 23331 sys_catalog.cc:564] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:30.741797 23334 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac006ac64c34290ab947dc9784f353b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 43927 } }
I20260926 07:01:30.742180 23335 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eac006ac64c34290ab947dc9784f353b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac006ac64c34290ab947dc9784f353b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 43927 } } }
I20260926 07:01:30.742280 23335 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:30.742197 23336 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac006ac64c34290ab947dc9784f353b. Latest consensus state: current_term: 1 leader_uuid: "eac006ac64c34290ab947dc9784f353b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac006ac64c34290ab947dc9784f353b" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 43927 } } }
I20260926 07:01:30.742408 23336 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:30.742538 23343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:30.743125 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 23263
I20260926 07:01:30.743145 23343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:30.743335 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:30.743588 1321 external_mini_cluster.cc:1428] Setting key 768f91852fb122b09d37df5408ee2c75
I20260926 07:01:30.745090 23343 catalog_manager.cc:1347] Generated new cluster ID: 68fccbb483b3465b819e5568cb8feb42
I20260926 07:01:30.745146 23343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:30.748811 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:30.744039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45057 (local address 127.1.74.126:43927)
0926 07:01:30.744213 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:30.744223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:30.744457 (+ 234us) server_negotiation.cc:407] Connection header received
0926 07:01:30.744586 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:30.744594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:30.744652 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:30.744816 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:30.745059 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:30.746022 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:30.746823 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:30.747301 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:30.747400 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:30.748123 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:30.748127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:30.748127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:30.748326 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:30.748355 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:30.748366 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:30.748443 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:30.748528 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 07:01:30.748590 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260926 07:01:30.749876 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:30.754083 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:30.751214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42535 (local address 127.1.74.126:43927)
0926 07:01:30.751249 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:30.751264 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:30.751387 (+ 123us) server_negotiation.cc:407] Connection header received
0926 07:01:30.751458 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:30.751459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:30.751470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:30.751517 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:30.751665 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:30.752131 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:30.752847 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:30.753059 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:30.753141 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:30.753803 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:30.753804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:30.753804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:30.753809 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:30.753822 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:30.753826 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:30.753879 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:30.753939 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:01:30.753962 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:01:30.773910 23343 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:30.774677 23343 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:30.779182 23343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eac006ac64c34290ab947dc9784f353b: Generated new TSK 0
I20260926 07:01:30.779387 23343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:30.784693 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 23263
2026-09-26T07:01:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:30 dist-test-slave-sfvt krb5kdc[23362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:30 dist-test-slave-sfvt krb5kdc[23362](info): set up 2 sockets
Sep 26 07:01:30 dist-test-slave-sfvt krb5kdc[23362](info): commencing operation
krb5kdc: starting...
W20260926 07:01:31.827414 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:31 dist-test-slave-sfvt krb5kdc[23362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:31.944368 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46041
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:46041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:32.023912 23375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:32.024183 23375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:32.024223 23375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:32.025389 23375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:32.025470 23375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:32.025492 23375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:32.025512 23375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:32.025552 23375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:32.026995 23375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:46041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:46041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:32.027300 23375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:32.027534 23375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:32.032462 23380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:32.032418 23381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:32.032425 23383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:32.032783 23375 server_base.cc:1034] running on GCE node
I20260926 07:01:32.033027 23375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:32.033308 23375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:32.034452 23375 hybrid_clock.cc:648] HybridClock initialized: now 1790406092034432 us; error 36 us; skew 500 ppm
Sep 26 07:01:32 dist-test-slave-sfvt krb5kdc[23362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:32.040540 23375 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:32.041000 23375 webserver.cc:460] Webserver started at http://127.1.74.126:40501/ using document root <none> and password file <none>
I20260926 07:01:32.041229 23375 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:32.041298 23375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:32.041451 23375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:32.042536 23375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33"
format_stamp: "Formatted at 2026-09-26 07:01:32 on dist-test-slave-sfvt"
server_key: "a61e26087a9f2b2d4b038c1d951c3031"
server_key_iv: "287bbefefaa9dec590a4a29b4e06c646"
server_key_version: "encryptionkey@0"
I20260926 07:01:32.042897 23375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33"
format_stamp: "Formatted at 2026-09-26 07:01:32 on dist-test-slave-sfvt"
server_key: "a61e26087a9f2b2d4b038c1d951c3031"
server_key_iv: "287bbefefaa9dec590a4a29b4e06c646"
server_key_version: "encryptionkey@0"
I20260926 07:01:32.044343 23375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:32.045114 23390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:32.045276 23375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:32.045377 23375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33"
format_stamp: "Formatted at 2026-09-26 07:01:32 on dist-test-slave-sfvt"
server_key: "a61e26087a9f2b2d4b038c1d951c3031"
server_key_iv: "287bbefefaa9dec590a4a29b4e06c646"
server_key_version: "encryptionkey@0"
I20260926 07:01:32.045466 23375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:32.048861 23375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:32.050642 23375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:32.050804 23375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:32.054759 23375 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:46041
I20260926 07:01:32.054803 23442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:46041 every 8 connection(s)
I20260926 07:01:32.055181 23375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:32.056252 23443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:32.058457 23443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33: Bootstrap starting.
I20260926 07:01:32.059008 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 23375
I20260926 07:01:32.059051 23443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:32.059116 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:32.059350 1321 external_mini_cluster.cc:1428] Setting key 8c340c2250b501076129a637bf361a1b
I20260926 07:01:32.059422 23443 log.cc:824] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:32.060192 23443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33: No bootstrap required, opened a new log
I20260926 07:01:32.061838 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:32.059839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45523 (local address 127.1.74.126:46041)
0926 07:01:32.059991 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:32.060000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:32.060039 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:01:32.060534 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:32.060541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:32.060574 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:32.060800 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:32.061392 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:32.061395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:32.061395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:32.061399 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:32.061434 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:32.061445 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:32.061515 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:32.061581 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 07:01:32.061613 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260926 07:01:32.062803 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:32.062747 23443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46041 } }
I20260926 07:01:32.062929 23443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:32.062968 23443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d3f4d8e6ab741fd839b4be9b49dbe33, State: Initialized, Role: FOLLOWER
I20260926 07:01:32.063078 23443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46041 } }
I20260926 07:01:32.063155 23443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:32.063184 23443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:32.063227 23443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:32.063984 23443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46041 } }
I20260926 07:01:32.064105 23443 leader_election.cc:302] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d3f4d8e6ab741fd839b4be9b49dbe33; no voters:
I20260926 07:01:32.064286 23443 leader_election.cc:288] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:32.064321 23452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:32.064462 23452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [term 1 LEADER]: Becoming Leader. State: Replica: 9d3f4d8e6ab741fd839b4be9b49dbe33, State: Running, Role: LEADER
I20260926 07:01:32.064548 23443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:32.064596 23452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46041 } }
I20260926 07:01:32.065008 23453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46041 } } }
I20260926 07:01:32.065200 23453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:32.064996 23452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d3f4d8e6ab741fd839b4be9b49dbe33. Latest consensus state: current_term: 1 leader_uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d3f4d8e6ab741fd839b4be9b49dbe33" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 46041 } } }
I20260926 07:01:32.065405 23452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:32.065513 23462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:32.065762 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:32.064419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36289 (local address 127.1.74.126:46041)
0926 07:01:32.064453 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:32.064455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:32.064690 (+ 235us) server_negotiation.cc:407] Connection header received
0926 07:01:32.064780 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:32.064781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:32.064785 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:32.064811 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:32.065434 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:32.065435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:32.065436 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:32.065440 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:32.065453 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:32.065457 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:32.065502 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:32.065676 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 07:01:32.065690 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:32.066017 23462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:32.067672 23462 catalog_manager.cc:1347] Generated new cluster ID: 847e2ba293784f2f8ecb846937542e41
I20260926 07:01:32.067725 23462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:32.080199 23462 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:32.080380 23462 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:32.093909 23462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d3f4d8e6ab741fd839b4be9b49dbe33: Generated new TSK 0
I20260926 07:01:32.094168 23462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:32.104393 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 23375
2026-09-26T07:01:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1319 ms)
[----------] 18 tests from AuthTokenIssuingTest (23766 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:32 dist-test-slave-sfvt krb5kdc[23474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:32 dist-test-slave-sfvt krb5kdc[23474](info): set up 2 sockets
Sep 26 07:01:32 dist-test-slave-sfvt krb5kdc[23474](info): commencing operation
krb5kdc: starting...
W20260926 07:01:33.135506 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:33.253986 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45771
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:38867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:45771,127.1.74.125:36399,127.1.74.124:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:33.335309 23487 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:01:33.335538 23487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:33.335556 23487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:33.335575 23487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:33.336730 23487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:33.336774 23487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:33.336789 23487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:33.336802 23487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:33.336856 23487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:33.338400 23487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:38867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:45771,127.1.74.125:36399,127.1.74.124:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:45771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:33.338647 23487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:33.338943 23487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:33.343931 23493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:33.343931 23495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:33.343946 23492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:33.344306 23487 server_base.cc:1034] running on GCE node
I20260926 07:01:33.344570 23487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:33.344808 23487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:33.345958 23487 hybrid_clock.cc:648] HybridClock initialized: now 1790406093345939 us; error 27 us; skew 500 ppm
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:33.348498 23487 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:33.348978 23487 webserver.cc:460] Webserver started at http://127.1.74.126:37177/ using document root <none> and password file <none>
I20260926 07:01:33.349242 23487 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:33.349310 23487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:33.349460 23487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:33.350517 23487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "5e06b01aa6bf4e5d8e841311db70b007"
format_stamp: "Formatted at 2026-09-26 07:01:33 on dist-test-slave-sfvt"
server_key: "f0857a71b55557cc92f9905df8733259"
server_key_iv: "e6a99e42962a576e3f6393d657ad2ed1"
server_key_version: "encryptionkey@0"
I20260926 07:01:33.350870 23487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "5e06b01aa6bf4e5d8e841311db70b007"
format_stamp: "Formatted at 2026-09-26 07:01:33 on dist-test-slave-sfvt"
server_key: "f0857a71b55557cc92f9905df8733259"
server_key_iv: "e6a99e42962a576e3f6393d657ad2ed1"
server_key_version: "encryptionkey@0"
I20260926 07:01:33.352223 23487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:33.352893 23502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:33.353062 23487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:33.353161 23487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "5e06b01aa6bf4e5d8e841311db70b007"
format_stamp: "Formatted at 2026-09-26 07:01:33 on dist-test-slave-sfvt"
server_key: "f0857a71b55557cc92f9905df8733259"
server_key_iv: "e6a99e42962a576e3f6393d657ad2ed1"
server_key_version: "encryptionkey@0"
I20260926 07:01:33.353237 23487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:33.370911 23487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:33.372807 23487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:33.372983 23487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:33.377215 23487 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:45771
I20260926 07:01:33.377252 23554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:45771 every 8 connection(s)
I20260926 07:01:33.377610 23487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:33.378557 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 23487
I20260926 07:01:33.378660 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:33.378690 23555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:33.378844 1321 external_mini_cluster.cc:1428] Setting key daaf505b9f7f7de6b8d3ba77d2591873
I20260926 07:01:33.380941 23555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.74.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.125@KRBTEST.COM" created.
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:33.393062 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.381655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45771 (local address 127.0.0.1:51276)
0926 07:01:33.381806 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.381840 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:33.382424 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:33.382725 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:33.382727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:33.383007 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:33.383157 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.383161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.384240 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.384241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.384899 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.384901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.385429 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.385430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.385496 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.386065 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:33.386073 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:33.389434 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:33.391627 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.391633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.391635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.391821 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.392105 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.392106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.392106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.392188 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.392634 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:33.392638 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:33.392694 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 07:01:33.392764 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 07:01:33.392889 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260926 07:01:33.393153 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.381841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.1.74.126:45771)
0926 07:01:33.381970 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.381975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.382017 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:01:33.382608 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.382610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.382626 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.382720 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.383284 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.384124 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.385007 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.385311 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.385446 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.389547 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.389554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.389555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.389561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.391551 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.391913 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.391916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.391917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.392030 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.392255 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.392256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.392257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.392563 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.392652 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.392833 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.392979 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260926 07:01:33.393927 23555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:33.394263 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.394155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:36399 (local address 127.0.0.1:55708)
0926 07:01:33.394201 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.394243 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 07:01:33.394428 23504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
I20260926 07:01:33.394868 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.394783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:36399 (local address 127.0.0.1:55710)
0926 07:01:33.394828 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.394852 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 07:01:33.394989 23555 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.125:36399: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab.
Entry for principal kudu/127.1.74.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab.
I20260926 07:01:33.406491 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.125:36399
--webserver_interface=127.1.74.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab
--principal=kudu/127.1.74.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:38867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:45771,127.1.74.125:36399,127.1.74.124:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:01:33.444199 23555 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } attempt: 1
I20260926 07:01:33.444681 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.444539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:36399 (local address 127.0.0.1:55716)
0926 07:01:33.444610 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.444652 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 07:01:33.445047 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.444951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:36399 (local address 127.0.0.1:55728)
0926 07:01:33.444997 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.445021 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 07:01:33.445128 23555 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.125:36399: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
W20260926 07:01:33.488548 23562 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:01:33.488843 23562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:33.488878 23562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:33.488925 23562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:33.490067 23562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:33.490139 23562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:33.490164 23562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:33.490192 23562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:33.490216 23562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:33.491756 23562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:38867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:45771,127.1.74.125:36399,127.1.74.124:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab
--principal=kudu/127.1.74.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.125:36399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.74.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:33.492048 23562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:33.492290 23562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:33.497277 23570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:33.497253 23567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:33.497452 23562 server_base.cc:1034] running on GCE node
W20260926 07:01:33.497300 23568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:33.497789 23562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:33.498096 23562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:33.499317 23562 hybrid_clock.cc:648] HybridClock initialized: now 1790406093499312 us; error 37 us; skew 500 ppm
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:33.501926 23562 init.cc:377] Logged in from keytab as kudu/127.1.74.125@KRBTEST.COM (short username kudu)
I20260926 07:01:33.502357 23562 webserver.cc:460] Webserver started at http://127.1.74.125:39183/ using document root <none> and password file <none>
I20260926 07:01:33.502595 23562 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:33.502674 23562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:33.502821 23562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:33.503940 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/data/instance:
uuid: "6eee2f154aa44ab89e7acef376da2a36"
format_stamp: "Formatted at 2026-09-26 07:01:33 on dist-test-slave-sfvt"
server_key: "c41fd0bf67bd66e1c7ca732b169759c4"
server_key_iv: "9dd5d9a1877b46203ee2a82b27e66045"
server_key_version: "encryptionkey@0"
I20260926 07:01:33.504293 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/wal/instance:
uuid: "6eee2f154aa44ab89e7acef376da2a36"
format_stamp: "Formatted at 2026-09-26 07:01:33 on dist-test-slave-sfvt"
server_key: "c41fd0bf67bd66e1c7ca732b169759c4"
server_key_iv: "9dd5d9a1877b46203ee2a82b27e66045"
server_key_version: "encryptionkey@0"
I20260926 07:01:33.505637 23562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:33.506376 23577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:33.506542 23562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:33.506640 23562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/wal
uuid: "6eee2f154aa44ab89e7acef376da2a36"
format_stamp: "Formatted at 2026-09-26 07:01:33 on dist-test-slave-sfvt"
server_key: "c41fd0bf67bd66e1c7ca732b169759c4"
server_key_iv: "9dd5d9a1877b46203ee2a82b27e66045"
server_key_version: "encryptionkey@0"
I20260926 07:01:33.506722 23562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:33.513276 23555 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } attempt: 2
I20260926 07:01:33.513711 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.513572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:36399 (local address 127.0.0.1:55732)
0926 07:01:33.513634 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.513677 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 07:01:33.514076 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.513993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:36399 (local address 127.0.0.1:55738)
0926 07:01:33.514034 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.514059 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 07:01:33.514196 23555 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.125:36399: Network error: Client connection negotiation failed: client connection to 127.1.74.125:36399: connect: Connection refused (error 111)
I20260926 07:01:33.529351 23562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:33.531515 23562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:33.531667 23562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:33.535742 23562 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.125:36399
I20260926 07:01:33.535817 23629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.125:36399 every 8 connection(s)
I20260926 07:01:33.536157 23562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
I20260926 07:01:33.537106 23630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:33.538591 23630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:33.541083 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 23562
I20260926 07:01:33.541203 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-1/wal/instance
I20260926 07:01:33.541425 1321 external_mini_cluster.cc:1428] Setting key ee35fa954d974ccbede059013cbd73ee
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:33.548101 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.539227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51282 (local address 127.1.74.126:45771)
0926 07:01:33.539308 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.539310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.539511 (+ 201us) server_negotiation.cc:407] Connection header received
0926 07:01:33.540000 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.540001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.540015 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.540069 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.540692 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.541220 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.542257 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.542574 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.542652 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.544912 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.544914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.544914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.544923 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.546626 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.547027 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.547029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.547029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.547094 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.547397 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.547398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.547399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.547556 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.547632 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.547898 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.547984 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:01:33.548118 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.539210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45771 (local address 127.0.0.1:51282)
0926 07:01:33.539387 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.539429 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:33.539897 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:33.540145 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:33.540149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:33.540410 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:33.540599 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.540604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.541329 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.541329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.542164 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.542167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.542699 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.542700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.542809 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.543384 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:33.543402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:33.544816 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:33.546717 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.546724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.546726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.546949 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.547163 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.547164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.547164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.547323 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.547626 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:33.547631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:33.547751 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 07:01:33.547830 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:01:33.547955 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260926 07:01:33.548722 23630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.74.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.124@KRBTEST.COM" created.
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 07:01:33.556046 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.548994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:36399 (local address 127.0.0.1:55750)
0926 07:01:33.549048 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.549060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:33.549106 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:33.549402 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:33.549403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:33.549539 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:33.549583 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.549584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.550518 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.550518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.551020 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.551022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.551612 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.551613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.551677 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.552101 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:33.552107 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:33.553292 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:33.555019 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.555020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.555020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.555181 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.555442 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.555443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.555443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.555497 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.555800 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:33.555801 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:33.555838 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:01:33.555906 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:01:33.555980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 07:01:33.556236 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.549073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.1.74.125:36399)
0926 07:01:33.549247 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.549251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.549264 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:01:33.549280 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.549298 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.549313 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.549403 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.549659 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.550413 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.551115 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.551521 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.551613 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.553389 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.553396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.553397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.553404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.554917 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.555320 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.555323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.555323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.555377 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.555560 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.555561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.555561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.555724 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.555824 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.555977 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.556074 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20260926 07:01:33.556798 23630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:33.557165 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.557028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:41952)
0926 07:01:33.557081 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.557134 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 07:01:33.557354 23581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
I20260926 07:01:33.557802 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.557688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:41954)
0926 07:01:33.557741 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.557771 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 07:01:33.557933 23630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:33999: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab.
Entry for principal kudu/127.1.74.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab.
I20260926 07:01:33.567203 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.124:33999
--webserver_interface=127.1.74.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab
--principal=kudu/127.1.74.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:38867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:45771,127.1.74.125:36399,127.1.74.124:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:01:33.605456 23555 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } attempt: 3
I20260926 07:01:33.607170 23630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } attempt: 1
I20260926 07:01:33.607601 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.607468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:41960)
0926 07:01:33.607522 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.607564 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 07:01:33.607966 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.607868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:41968)
0926 07:01:33.607917 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.607945 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 07:01:33.608152 23630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:33999: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 07:01:33.612932 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.605972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.1.74.125:36399)
0926 07:01:33.606043 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.606045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.606055 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:33.606084 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.606085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.606097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.606162 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.606506 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.607006 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.608010 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.608334 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.608422 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.610352 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.610354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.610354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.610361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.611799 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.612155 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.612156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.612157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.612212 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.612423 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.612424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.612425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.612541 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.612615 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.612762 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.612840 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:01:33.612936 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.605872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:36399 (local address 127.0.0.1:55758)
0926 07:01:33.605946 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.605964 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:33.606026 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:33.606183 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:33.606185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:33.606353 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:33.606422 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.606424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.607105 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.607107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.607908 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.607912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.608423 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.608424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.608502 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.608949 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:33.608955 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:33.610247 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:33.611899 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.611900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.611900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.612071 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.612290 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.612291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.612291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.612353 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.612605 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:33.612606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:33.612651 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:01:33.612738 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:01:33.612840 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 07:01:33.613485 23555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:33.613819 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.613721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:41980)
0926 07:01:33.613772 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.613801 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 07:01:33.614109 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.614028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:41992)
0926 07:01:33.614069 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.614091 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 07:01:33.614200 23555 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:33999: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
I20260926 07:01:33.645349 23555 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } attempt: 1
I20260926 07:01:33.645826 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.645695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:41996)
0926 07:01:33.645755 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.645796 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 07:01:33.646183 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.646061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:42002)
0926 07:01:33.646105 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.646159 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 07:01:33.646272 23555 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:33999: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
W20260926 07:01:33.649681 23637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:01:33.649866 23637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:33.649883 23637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:33.649948 23637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:33.651090 23637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:33.651125 23637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:33.651139 23637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:33.651181 23637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:33.651201 23637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:33.652722 23637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:38867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:45771,127.1.74.125:36399,127.1.74.124:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab
--principal=kudu/127.1.74.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.124:33999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.74.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:33.652940 23637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:33.653151 23637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:33.657968 23643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:33.657976 23645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:33.657986 23637 server_base.cc:1034] running on GCE node
W20260926 07:01:33.657971 23642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:33.658535 23637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:33.658823 23637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:33.659991 23637 hybrid_clock.cc:648] HybridClock initialized: now 1790406093659983 us; error 38 us; skew 500 ppm
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:33.662421 23637 init.cc:377] Logged in from keytab as kudu/127.1.74.124@KRBTEST.COM (short username kudu)
I20260926 07:01:33.662788 23637 webserver.cc:460] Webserver started at http://127.1.74.124:46421/ using document root <none> and password file <none>
I20260926 07:01:33.662959 23637 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:33.662998 23637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:33.663085 23637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:33.664175 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/data/instance:
uuid: "9ad3d97ba71e423cbcd567630a6fc67a"
format_stamp: "Formatted at 2026-09-26 07:01:33 on dist-test-slave-sfvt"
server_key: "5ef8380a0c0b92f2a3bcf96accd0d3bb"
server_key_iv: "509169d51a68f5f7d8a3659565da86f7"
server_key_version: "encryptionkey@0"
I20260926 07:01:33.664494 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/wal/instance:
uuid: "9ad3d97ba71e423cbcd567630a6fc67a"
format_stamp: "Formatted at 2026-09-26 07:01:33 on dist-test-slave-sfvt"
server_key: "5ef8380a0c0b92f2a3bcf96accd0d3bb"
server_key_iv: "509169d51a68f5f7d8a3659565da86f7"
server_key_version: "encryptionkey@0"
I20260926 07:01:33.665702 23637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:33.666371 23652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:33.666522 23637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:33.666604 23637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/wal
uuid: "9ad3d97ba71e423cbcd567630a6fc67a"
format_stamp: "Formatted at 2026-09-26 07:01:33 on dist-test-slave-sfvt"
server_key: "5ef8380a0c0b92f2a3bcf96accd0d3bb"
server_key_iv: "509169d51a68f5f7d8a3659565da86f7"
server_key_version: "encryptionkey@0"
I20260926 07:01:33.666657 23637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:33.676301 23630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } attempt: 2
I20260926 07:01:33.676743 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.676604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:42010)
0926 07:01:33.676665 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.676709 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 07:01:33.677063 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.676997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:42026)
0926 07:01:33.677022 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.677045 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 07:01:33.677181 23630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:33999: Network error: Client connection negotiation failed: client connection to 127.1.74.124:33999: connect: Connection refused (error 111)
I20260926 07:01:33.684455 23637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:33.686278 23637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:33.686432 23637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:33.690344 23637 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.124:33999
I20260926 07:01:33.690405 23704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.124:33999 every 8 connection(s)
I20260926 07:01:33.690755 23637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
I20260926 07:01:33.691725 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 23637
I20260926 07:01:33.691835 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790405833506994-1321-0/minicluster-data/master-2/wal/instance
I20260926 07:01:33.691902 23705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:33.692041 1321 external_mini_cluster.cc:1428] Setting key 74d212202621b8d88996d340e6faf991
I20260926 07:01:33.693938 23705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:33.702565 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.694593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.1.74.126:45771)
0926 07:01:33.694662 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.694664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.694800 (+ 136us) server_negotiation.cc:407] Connection header received
0926 07:01:33.694878 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.694879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.694888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.694936 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.695485 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.696217 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.697254 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.697499 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.697567 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.699860 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.699862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.699862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.699870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.701120 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.701759 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.701761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.701761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.701822 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.702017 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.702017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.702018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.702129 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.702203 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.702396 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.702470 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:01:33.702684 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.694529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:45771 (local address 127.0.0.1:51292)
0926 07:01:33.694694 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.694734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:33.694817 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:33.694966 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:33.694969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:33.695330 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:33.695413 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.695415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.696308 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.696309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.697167 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.697169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.697603 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.697604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.697656 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.698105 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:33.698113 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:33.699771 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:33.701451 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.701456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.701458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.701664 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.701889 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.701890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.701890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.701953 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.702195 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:33.702201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:33.702257 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 07:01:33.702355 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:01:33.702463 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260926 07:01:33.702684 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.692539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.1.74.124:33999)
0926 07:01:33.692734 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.692743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.692779 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:01:33.693226 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.693233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.693278 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.693426 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.694058 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.695333 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.696396 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.696966 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.697070 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.699368 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.699374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.699374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.699379 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.701116 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.701449 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.701451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.701452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.701544 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.701778 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.701779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.701779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.702009 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.702158 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.702259 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.702381 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260926 07:01:33.703326 23705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:33.710237 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.703628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.1.74.125:36399)
0926 07:01:33.703678 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.703680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.703850 (+ 170us) server_negotiation.cc:407] Connection header received
0926 07:01:33.703876 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.703877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.703888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.703957 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.704159 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.704683 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.705353 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.705612 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.705691 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.707414 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.707415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.707416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.707422 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.709030 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.709373 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.709374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.709375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.709435 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.709657 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.709658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.709658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.709812 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.709891 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.710055 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.710139 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:01:33.710238 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.703576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:36399 (local address 127.0.0.1:55764)
0926 07:01:33.703774 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.703782 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:33.703813 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:33.703957 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:33.703957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:33.704065 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:33.704093 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.704094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.704762 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.704762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.705274 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.705276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.705704 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.705705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.705774 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.706189 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:33.706192 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:33.707338 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:33.709131 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.709132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.709132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.709305 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.709504 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.709505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.709505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.709586 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.709899 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:33.709900 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:33.709949 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 07:01:33.710020 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:01:33.710106 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260926 07:01:33.710976 23705 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:33.711402 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.703565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.1.74.126:45771)
0926 07:01:33.703662 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.703664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.703678 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:01:33.703694 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.703694 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.703701 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.703741 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.704326 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.704777 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.705438 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.705606 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.705653 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.708063 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.708064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.708065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.708071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.709664 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.710028 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.710029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.710029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.710084 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.710718 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.710719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.710719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.710872 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.711043 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.711083 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.711161 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 07:01:33.718509 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.711220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:42038)
0926 07:01:33.711309 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.711319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:33.711355 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:33.711493 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:33.711494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:33.711637 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:33.711698 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.711701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.712599 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.712600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.713063 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.713064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.713460 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.713461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.713507 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.713935 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:33.713939 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:33.715151 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:33.717333 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.717334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.717335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.717487 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.717683 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.717683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.717683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.717730 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.718180 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:33.718180 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:33.718214 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:01:33.718328 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:01:33.718430 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 07:01:33.718638 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.711296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.1.74.124:33999)
0926 07:01:33.711330 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.711332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.711343 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:01:33.711402 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.711403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.711409 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.711485 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.711784 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.712526 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.713163 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.713379 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.713461 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.715276 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.715278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.715278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.715285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.717254 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.717560 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.717561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.717562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.717615 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.717783 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.717784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.717784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.717909 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.717972 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.718360 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.718491 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":19}
I20260926 07:01:33.719228 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.712037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.1.74.125:36399)
0926 07:01:33.712070 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.712072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.712079 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:01:33.712099 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.712100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.712107 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.712153 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.712684 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.713118 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.713840 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.714138 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.714211 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.716505 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.716506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.716506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.716512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.717901 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.718179 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.718180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.718180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.718224 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.718436 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.718437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.718437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.718560 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.718806 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.719010 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.719085 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 07:01:33.720212 23705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a: Bootstrap starting.
I20260926 07:01:33.720685 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:33.721170 23705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:33.721387 23555 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } attempt: 2
I20260926 07:01:33.721487 23705 log.cc:824] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:33.722565 23705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a: No bootstrap required, opened a new log
I20260926 07:01:33.721799 23529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5e06b01aa6bf4e5d8e841311db70b007"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51304
I20260926 07:01:33.724571 23705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } }
I20260926 07:01:33.724727 23705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad3d97ba71e423cbcd567630a6fc67a, State: Initialized, Role: FOLLOWER
I20260926 07:01:33.724817 23705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } }
I20260926 07:01:33.725101 23711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:33.725167 23705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
I20260926 07:01:33.725225 23711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [sys.catalog]: This master's current role is: FOLLOWER
W20260926 07:01:33.727069 23722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:33.727133 23722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:33.728493 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.721682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.1.74.124:33999)
0926 07:01:33.721715 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.721717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.721740 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:01:33.721834 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.721834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.721844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.721893 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.722209 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.722649 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.723477 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.723678 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.723759 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.725657 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.725659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.725659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.725667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.727339 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.727697 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.727698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.727699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.727754 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.727939 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.727939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.727940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.728102 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.728165 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.728333 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.728404 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:33.728494 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.721653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:42048)
0926 07:01:33.721699 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.721709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:33.721754 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:33.721918 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:33.721919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:33.722072 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:33.722128 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.722130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.722749 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.722751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.723397 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.723400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.723761 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.723762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.723819 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.724240 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:33.724245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:33.725564 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:33.727427 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.727428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.727429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.727604 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.727824 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.727824 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.727825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.727881 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.728156 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:33.728157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:33.728219 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 07:01:33.728308 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:01:33.728405 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 07:01:33.729681 23555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007: Bootstrap starting.
I20260926 07:01:33.730288 23555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:33.727563 23637 master_runner.cc:186] Error getting master registration for 127.1.74.126:45771: OK, instance_id { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" instance_seqno: 1790406093376072 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:01:33.730356 23637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:01:33.730597 23555 log.cc:824] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:33.731388 23555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007: No bootstrap required, opened a new log
I20260926 07:01:33.732476 23555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } }
I20260926 07:01:33.732606 23555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e06b01aa6bf4e5d8e841311db70b007, State: Initialized, Role: FOLLOWER
I20260926 07:01:33.732692 23555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } }
I20260926 07:01:33.733027 23555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:33.733062 23529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:01:33.733103 23529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:33.733918 23529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } }
I20260926 07:01:33.734158 23529 leader_election.cc:288] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [CANDIDATE]: Term 1 election: Requested vote from peers 6eee2f154aa44ab89e7acef376da2a36 (127.1.74.125:36399), 9ad3d97ba71e423cbcd567630a6fc67a (127.1.74.124:33999)
W20260926 07:01:33.734863 23736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:33.734937 23736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:33.733002 23725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:33.735628 23725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:33.735803 23679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e06b01aa6bf4e5d8e841311db70b007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ad3d97ba71e423cbcd567630a6fc67a"
I20260926 07:01:33.735910 23679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:33.737100 23679 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e06b01aa6bf4e5d8e841311db70b007 in term 1.
I20260926 07:01:33.737385 23506 leader_election.cc:302] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e06b01aa6bf4e5d8e841311db70b007, 9ad3d97ba71e423cbcd567630a6fc67a; no voters:
I20260926 07:01:33.737496 23725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:33.737628 23725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [term 1 LEADER]: Becoming Leader. State: Replica: 5e06b01aa6bf4e5d8e841311db70b007, State: Running, Role: LEADER
I20260926 07:01:33.737713 23725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } }
I20260926 07:01:33.738106 23725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e06b01aa6bf4e5d8e841311db70b007. Latest consensus state: current_term: 1 leader_uuid: "5e06b01aa6bf4e5d8e841311db70b007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:33.738171 23725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:33.738334 23744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:33.738523 23744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:33.737644 23487 master_runner.cc:186] Error getting master registration for 127.1.74.125:36399: OK, instance_id { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" instance_seqno: 1790406093534480 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:01:33.740363 23487 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:01:33.741933 23679 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [term 1 FOLLOWER]: Refusing update from remote peer 5e06b01aa6bf4e5d8e841311db70b007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:33.742352 23725 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:33.743798 23711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e06b01aa6bf4e5d8e841311db70b007. Latest consensus state: current_term: 1 leader_uuid: "5e06b01aa6bf4e5d8e841311db70b007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:33.743891 23711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:33.745922 23742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e06b01aa6bf4e5d8e841311db70b007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:33.746017 23742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:33.746429 23711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e06b01aa6bf4e5d8e841311db70b007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:33.746510 23711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:33.746762 23748 mvcc.cc:205] Tried to move back new op lower bound from 7333503359964393472 to 7333503359951208448. Current Snapshot: MvccSnapshot[applied={T|T < 7333503359964393472}]
I20260926 07:01:33.747713 23749 mvcc.cc:205] Tried to move back new op lower bound from 7333503359964393472 to 7333503359951208448. Current Snapshot: MvccSnapshot[applied={T|T < 7333503359964393472}]
I20260926 07:01:33.748597 23744 catalog_manager.cc:1347] Generated new cluster ID: a8988cfe02ee4a63a7708c2e4d1587f5
I20260926 07:01:33.748723 23744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:33.759619 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.739440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.1.74.126:45771)
0926 07:01:33.739475 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.739478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.740278 (+ 800us) server_negotiation.cc:407] Connection header received
0926 07:01:33.740900 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.740900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.740910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.740939 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.742974 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.743725 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.749701 (+ 5976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.749998 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.750033 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.754157 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.754159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.754159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.754165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.755764 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.758191 (+ 2427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.758192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.758193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.758249 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.758426 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.758426 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.758427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.758547 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.758616 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.759440 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.759521 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:01:33.762137 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.739818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.1.74.124:33999)
0926 07:01:33.739854 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.739856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.740790 (+ 934us) server_negotiation.cc:407] Connection header received
0926 07:01:33.741861 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.741862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.741871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.741896 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.745512 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.746176 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.748832 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.749102 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.749146 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.756058 (+ 6912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.756060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.756060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.756068 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.757666 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.758668 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.758669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.758669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.758723 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.759029 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.759030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.759031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.759149 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.759228 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.761973 (+ 2745us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.762049 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:33.763962 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.739965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.1.74.125:36399)
0926 07:01:33.740004 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.740006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.742779 (+ 2773us) server_negotiation.cc:407] Connection header received
0926 07:01:33.742875 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.742876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.742887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.742915 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.747151 (+ 4236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.747850 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.752469 (+ 4619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.752785 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.752826 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.759951 (+ 7125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.759952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.759952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.759960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.761579 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.763092 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.763093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.763093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.763153 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.763402 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.763403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.763403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.763532 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.763604 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.763789 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.763864 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:01:33.768333 23630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } attempt: 3
Sep 26 07:01:33 dist-test-slave-sfvt krb5kdc[23474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406093, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
I20260926 07:01:33.776232 23744 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:33.777015 23744 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:33.777112 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.768616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:33999 (local address 127.0.0.1:42066)
0926 07:01:33.768705 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:33.768716 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:33.768754 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:33.768958 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:33.768958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:33.769129 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:33.769171 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.769173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.770062 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.770063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.770864 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:33.770866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.771394 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.771395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:33.771452 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.772130 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:33.772135 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:33.773663 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:33.775915 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.775916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.775917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.776131 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.776432 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:33.776433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:33.776433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:33.776513 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:33.776818 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:33.776819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:33.776874 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 07:01:33.776934 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:01:33.777024 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 07:01:33.777526 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.768820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.1.74.124:33999)
0926 07:01:33.768866 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.768868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.768874 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:01:33.768894 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.768895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.768904 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.768955 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.769252 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.769976 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.770954 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.771318 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.771397 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.773751 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:33.773752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:33.773752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:33.773759 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:33.775594 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.776217 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.776218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.776219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.776274 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:33.776593 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:33.776594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:33.776595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:33.776725 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:33.776776 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.777367 (+ 591us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.777439 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 07:01:33.779098 23630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36: Bootstrap starting.
I20260926 07:01:33.779908 23630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:33.780186 23630 log.cc:824] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:33.781541 23630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36: No bootstrap required, opened a new log
I20260926 07:01:33.784765 23630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } }
I20260926 07:01:33.784909 23630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eee2f154aa44ab89e7acef376da2a36, State: Initialized, Role: FOLLOWER
I20260926 07:01:33.784994 23630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } }
I20260926 07:01:33.785364 23744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007: Generated new TSK 0
I20260926 07:01:33.785305 23754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:33.785545 23744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:33.785610 23754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:33.785387 23602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:33.785411 23604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e06b01aa6bf4e5d8e841311db70b007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6eee2f154aa44ab89e7acef376da2a36"
I20260926 07:01:33.786533 23602 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [term 1 FOLLOWER]: Refusing update from remote peer 5e06b01aa6bf4e5d8e841311db70b007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:33.786904 23742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:33.787492 23604 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e06b01aa6bf4e5d8e841311db70b007 in term 1.
I20260926 07:01:33.785334 23630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:33.789177 23757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e06b01aa6bf4e5d8e841311db70b007. Latest consensus state: current_term: 1 leader_uuid: "5e06b01aa6bf4e5d8e841311db70b007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:33.789324 23757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:33.789417 23757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e06b01aa6bf4e5d8e841311db70b007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:33.789631 23757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:33.790704 23771 catalog_manager.cc:1259] Loaded cluster ID: a8988cfe02ee4a63a7708c2e4d1587f5
I20260926 07:01:33.790787 23771 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36: loading cluster ID for follower catalog manager: success
I20260926 07:01:33.791978 23771 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36: acquiring CA information for follower catalog manager: success
I20260926 07:01:33.792624 23771 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6eee2f154aa44ab89e7acef376da2a36: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:01:33.814987 23559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.813474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.1.74.126:45771)
0926 07:01:33.813521 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.813523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.814286 (+ 763us) server_negotiation.cc:407] Connection header received
0926 07:01:33.814413 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.814415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.814426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.814456 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.814913 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51324: BlockingRecv error: recv got EOF from 127.0.0.1:51324 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 07:01:33.815766 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.813819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.1.74.124:33999)
0926 07:01:33.813870 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.813872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.815232 (+ 1360us) server_negotiation.cc:407] Connection header received
0926 07:01:33.815349 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.815350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.815363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.815392 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.815704 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42080: BlockingRecv error: recv got EOF from 127.0.0.1:42080 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 07:01:33.816577 23634 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.814154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.1.74.125:36399)
0926 07:01:33.814206 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.814208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.815984 (+ 1776us) server_negotiation.cc:407] Connection header received
0926 07:01:33.816054 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.816056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.816068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.816118 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.816478 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55788: BlockingRecv error: recv got EOF from 127.0.0.1:55788 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:01:33.821353 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.817784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55804 (local address 127.1.74.125:36399)
0926 07:01:33.817820 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.817822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.818018 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:01:33.818118 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.818118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.818159 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.818205 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:33.818389 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.819201 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.820376 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.820686 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.820726 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.820986 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:33.821110 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:33.821157 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.821268 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.821285 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 07:01:33.824530 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.823773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.1.74.124:33999)
0926 07:01:33.823809 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.823810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.824022 (+ 212us) server_negotiation.cc:407] Connection header received
0926 07:01:33.824126 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.824127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.824143 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.824172 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.824483 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42086: BlockingRecv error: recv got EOF from 127.0.0.1:42086 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:01:33.830121 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.826870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.1.74.125:36399)
0926 07:01:33.826904 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.826906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.827118 (+ 212us) server_negotiation.cc:407] Connection header received
0926 07:01:33.827208 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.827209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.827218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.827341 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:33.827510 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.828253 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.829343 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.829640 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.829690 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.829879 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:33.829911 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:33.829937 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.830037 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.830053 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 07:01:33.840034 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.835611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.1.74.124:33999)
0926 07:01:33.835661 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.835662 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.839496 (+ 3834us) server_negotiation.cc:407] Connection header received
0926 07:01:33.839597 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.839598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.839609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.839637 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.839979 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42102: BlockingRecv error: recv got EOF from 127.0.0.1:42102 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:33.846292 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.843094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.1.74.125:36399)
0926 07:01:33.843129 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.843131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.843403 (+ 272us) server_negotiation.cc:407] Connection header received
0926 07:01:33.843502 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.843503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.843512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.843540 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:33.843715 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.844445 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.845533 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.845831 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.845870 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.846050 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:33.846082 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:33.846107 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.846207 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.846223 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 07:01:33.849202 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.848484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.1.74.124:33999)
0926 07:01:33.848519 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.848533 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.848729 (+ 196us) server_negotiation.cc:407] Connection header received
0926 07:01:33.848819 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.848820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.848829 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.848857 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.849157 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42106: BlockingRecv error: recv got EOF from 127.0.0.1:42106 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:33.857604 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.854286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.1.74.125:36399)
0926 07:01:33.854322 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.854324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.854551 (+ 227us) server_negotiation.cc:407] Connection header received
0926 07:01:33.854675 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.854676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.854685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.854713 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:33.854874 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.855629 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.856817 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.857098 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.857140 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.857346 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:33.857379 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:33.857406 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.857508 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.857525 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 07:01:33.860584 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.859851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.1.74.124:33999)
0926 07:01:33.859888 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.859890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.860055 (+ 165us) server_negotiation.cc:407] Connection header received
0926 07:01:33.860156 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.860157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.860167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.860195 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.860530 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42108: BlockingRecv error: recv got EOF from 127.0.0.1:42108 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:33.873071 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.869834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.1.74.125:36399)
0926 07:01:33.869889 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.869891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.870165 (+ 274us) server_negotiation.cc:407] Connection header received
0926 07:01:33.870261 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.870262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.870271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.870299 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:33.870464 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.871209 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.872317 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.872605 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.872654 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.872831 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:33.872862 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:33.872887 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.872980 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.872996 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 07:01:33.876232 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.875471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.1.74.124:33999)
0926 07:01:33.875511 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.875513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.875732 (+ 219us) server_negotiation.cc:407] Connection header received
0926 07:01:33.875819 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.875819 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.875830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.875857 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.876183 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42112: BlockingRecv error: recv got EOF from 127.0.0.1:42112 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:33.897104 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.893541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.1.74.125:36399)
0926 07:01:33.893585 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.893587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.894054 (+ 467us) server_negotiation.cc:407] Connection header received
0926 07:01:33.894174 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.894175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.894186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.894217 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:33.894418 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.895158 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.896289 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.896563 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.896610 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.896832 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:33.896867 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:33.896895 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.897003 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.897023 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 07:01:33.900396 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.899539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.1.74.124:33999)
0926 07:01:33.899581 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.899583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.899832 (+ 249us) server_negotiation.cc:407] Connection header received
0926 07:01:33.899930 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.899931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.899943 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.899973 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.900337 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42124: BlockingRecv error: recv got EOF from 127.0.0.1:42124 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:01:33.936730 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:33.933928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.1.74.125:36399)
0926 07:01:33.933983 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.933985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.934137 (+ 152us) server_negotiation.cc:407] Connection header received
0926 07:01:33.934238 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.934239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.934251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.934310 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:33.934442 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.935185 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.935991 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:33.936293 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:33.936373 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:33.936494 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:33.936529 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:33.936582 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:33.936629 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 07:01:33.936647 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 07:01:33.939006 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:33.938453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.1.74.124:33999)
0926 07:01:33.938499 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:33.938501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:33.938510 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:33.938576 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:33.938577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:33.938587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:33.938627 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:33.938951 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42138: BlockingRecv error: recv got EOF from 127.0.0.1:42138 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:01:34.006628 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:34.004253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.1.74.125:36399)
0926 07:01:34.004322 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.004325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.004427 (+ 102us) server_negotiation.cc:407] Connection header received
0926 07:01:34.004476 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.004477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.004485 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.004534 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:34.004656 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.005296 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.006001 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.006180 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.006227 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:34.006359 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:34.006393 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:34.006444 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:34.006491 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 07:01:34.006507 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 07:01:34.008833 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:34.008288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.1.74.124:33999)
0926 07:01:34.008351 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.008355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.008371 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:01:34.008448 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.008449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.008459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.008511 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:34.008764 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42148: BlockingRecv error: recv got EOF from 127.0.0.1:42148 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:01:34.140108 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:34.137963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.1.74.125:36399)
0926 07:01:34.138044 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.138046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.138122 (+ 76us) server_negotiation.cc:407] Connection header received
0926 07:01:34.138167 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.138168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.138176 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.138239 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:34.138368 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.138826 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.139576 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.139748 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.139798 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:34.139908 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:34.139933 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:34.139969 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:34.140032 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:01:34.140047 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 07:01:34.142443 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:34.141799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.1.74.124:33999)
0926 07:01:34.141874 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.141876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.141903 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:01:34.141984 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.141985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.142004 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.142048 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:34.142358 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42164: BlockingRecv error: recv got EOF from 127.0.0.1:42164 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:01:34.227463 23725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e06b01aa6bf4e5d8e841311db70b007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e06b01aa6bf4e5d8e841311db70b007" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 45771 } } peers { permanent_uuid: "6eee2f154aa44ab89e7acef376da2a36" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 36399 } } peers { permanent_uuid: "9ad3d97ba71e423cbcd567630a6fc67a" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 33999 } } }
I20260926 07:01:34.227687 23725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e06b01aa6bf4e5d8e841311db70b007 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:34.401700 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:34.399619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.1.74.125:36399)
0926 07:01:34.399690 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.399692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.399703 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:01:34.399745 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.399746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.399755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.399818 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:34.399956 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.400417 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.401143 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.401317 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.401364 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:34.401497 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:34.401522 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:34.401559 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:34.401625 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 07:01:34.401641 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 07:01:34.404151 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:34.403532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.1.74.124:33999)
0926 07:01:34.403607 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.403609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.403618 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:34.403677 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.403678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.403688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.403727 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:34.404078 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42664: BlockingRecv error: recv got EOF from 127.0.0.1:42664 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:01:34.727591 23722 catalog_manager.cc:1259] Loaded cluster ID: a8988cfe02ee4a63a7708c2e4d1587f5
I20260926 07:01:34.727664 23722 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a: loading cluster ID for follower catalog manager: success
I20260926 07:01:34.728542 23722 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a: acquiring CA information for follower catalog manager: success
I20260926 07:01:34.728814 23722 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ad3d97ba71e423cbcd567630a6fc67a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:01:34.919632 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:34.917208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.1.74.125:36399)
0926 07:01:34.917335 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.917337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.917348 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:01:34.917411 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.917412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.917423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.917495 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:34.917629 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.918170 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.918989 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.919178 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.919237 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:34.919414 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:34.919442 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:34.919484 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:34.919544 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:01:34.919563 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260926 07:01:34.923918 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:34.921351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.1.74.124:33999)
0926 07:01:34.921462 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.921464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.921476 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:01:34.921504 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.921505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.921580 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.921665 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:34.921784 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.922280 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.923036 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.923310 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.923400 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:34.923575 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:34.923694 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:34.923786 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:34.923806 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 07:01:34.923827 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260926 07:01:34.928444 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:34.925602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.1.74.125:36399)
0926 07:01:34.925660 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.925662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.925886 (+ 224us) server_negotiation.cc:407] Connection header received
0926 07:01:34.925953 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.925954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.925966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.926014 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:34.926193 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.926913 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.927802 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.928049 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.928117 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:34.928221 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:34.928251 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:34.928303 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:34.928338 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 07:01:34.928363 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:01:34.928769 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:34.925711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.1.74.124:33999)
0926 07:01:34.925768 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.925770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.925906 (+ 136us) server_negotiation.cc:407] Connection header received
0926 07:01:34.926030 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.926031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.926042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.926072 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:34.926267 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.926805 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.928012 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.928306 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.928367 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:34.928495 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:34.928526 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:34.928577 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:34.928675 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 07:01:34.928695 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:01:34.929457 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:34.925579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.1.74.126:45771)
0926 07:01:34.925688 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:34.925690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:34.925826 (+ 136us) server_negotiation.cc:407] Connection header received
0926 07:01:34.925868 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:34.925868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:34.925919 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:34.926000 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:34.926091 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.926893 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.928490 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:34.928784 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:34.928845 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:34.929053 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:34.929169 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:34.929235 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:34.929376 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 07:01:34.929399 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":66,"threads_started":1}
I20260926 07:01:34.931139 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 23487
I20260926 07:01:34.936542 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 23562
I20260926 07:01:34.941469 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 23637
2026-09-26T07:01:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2838 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:34 dist-test-slave-sfvt krb5kdc[23904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:34 dist-test-slave-sfvt krb5kdc[23904](info): set up 2 sockets
Sep 26 07:01:34 dist-test-slave-sfvt krb5kdc[23904](info): commencing operation
krb5kdc: starting...
W20260926 07:01:35.967595 1321 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/test-user.keytab.
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.74.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
Entry for principal kudu/127.1.74.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab.
I20260926 07:01:36.082106 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:44321
--webserver_interface=127.1.74.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:44321,127.1.74.125:38339,127.1.74.124:41787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:36.162333 23917 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:01:36.162550 23917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:36.162568 23917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:36.162588 23917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:36.163797 23917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:36.163836 23917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:36.163851 23917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:36.163863 23917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:36.163923 23917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:36.165370 23917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:44321,127.1.74.125:38339,127.1.74.124:41787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.126.keytab
--principal=kudu/127.1.74.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.126:44321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.74.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:36.165621 23917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:36.165853 23917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:36.170840 23923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:36.170807 23922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:36.171113 23917 server_base.cc:1034] running on GCE node
W20260926 07:01:36.170826 23925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:36.171443 23917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:36.171692 23917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:36.172827 23917 hybrid_clock.cc:648] HybridClock initialized: now 1790406096172807 us; error 37 us; skew 500 ppm
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:36.175403 23917 init.cc:377] Logged in from keytab as kudu/127.1.74.126@KRBTEST.COM (short username kudu)
I20260926 07:01:36.175844 23917 webserver.cc:460] Webserver started at http://127.1.74.126:38423/ using document root <none> and password file <none>
I20260926 07:01:36.176041 23917 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:36.176082 23917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:36.176168 23917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:36.177233 23917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/data/instance:
uuid: "446f7ced7cfd450db0a42f7d450d3c2d"
format_stamp: "Formatted at 2026-09-26 07:01:36 on dist-test-slave-sfvt"
server_key: "661c09a879cfe4bddf43594575d77836"
server_key_iv: "e2becb64b2853576621f93f557979bda"
server_key_version: "encryptionkey@0"
I20260926 07:01:36.177532 23917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/wal/instance:
uuid: "446f7ced7cfd450db0a42f7d450d3c2d"
format_stamp: "Formatted at 2026-09-26 07:01:36 on dist-test-slave-sfvt"
server_key: "661c09a879cfe4bddf43594575d77836"
server_key_iv: "e2becb64b2853576621f93f557979bda"
server_key_version: "encryptionkey@0"
I20260926 07:01:36.178740 23917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:36.179436 23932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:36.179634 23917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:36.179731 23917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/wal
uuid: "446f7ced7cfd450db0a42f7d450d3c2d"
format_stamp: "Formatted at 2026-09-26 07:01:36 on dist-test-slave-sfvt"
server_key: "661c09a879cfe4bddf43594575d77836"
server_key_iv: "e2becb64b2853576621f93f557979bda"
server_key_version: "encryptionkey@0"
I20260926 07:01:36.179813 23917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:36.205719 23917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:36.208228 23917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:36.208398 23917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:36.212711 23917 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.126:44321
I20260926 07:01:36.212747 23984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.126:44321 every 8 connection(s)
I20260926 07:01:36.213135 23917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:36.214098 23985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:36.215703 23985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:36.216895 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 23917
I20260926 07:01:36.216987 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-0/wal/instance
I20260926 07:01:36.217204 1321 external_mini_cluster.cc:1428] Setting key 4c36238253e5ce97f569736f5ffd521c
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.74.125@KRBTEST.COM; defaulting to no policy
I20260926 07:01:36.226490 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.216439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.1.74.126:44321)
0926 07:01:36.216628 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.216633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.216672 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:01:36.217291 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.217294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.217320 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.217418 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.217961 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.219201 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.220101 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.220663 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.220774 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.223114 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.223123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.223124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.223131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.225016 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.225373 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.225376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.225376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.225458 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.225653 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.225654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.225655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.225853 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.225991 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.226250 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.226369 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260926 07:01:36.226496 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.216380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44321 (local address 127.0.0.1:52738)
0926 07:01:36.216540 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.216586 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:36.217134 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:36.217423 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:36.217426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:36.217714 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:36.217862 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.217867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.219336 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.219337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.220012 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.220014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.220821 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.220823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.220889 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.221457 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:36.221467 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:36.223022 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:36.225097 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.225103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.225105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.225289 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.225521 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.225522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.225522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.225597 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.225992 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:36.225997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:36.226068 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 07:01:36.226165 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:01:36.226296 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260926 07:01:36.227421 23985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:36.227829 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.227667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:38339 (local address 127.0.0.1:38764)
0926 07:01:36.227735 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.227786 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.125@KRBTEST.COM" created.
W20260926 07:01:36.227980 23935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
I20260926 07:01:36.229441 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.229323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:38339 (local address 127.0.0.1:38778)
0926 07:01:36.229378 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.229412 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 07:01:36.229629 23985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.125:38339: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab.
Entry for principal kudu/127.1.74.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab.
I20260926 07:01:36.242939 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.125:38339
--webserver_interface=127.1.74.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab
--principal=kudu/127.1.74.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:44321,127.1.74.125:38339,127.1.74.124:41787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:01:36.278883 23985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } attempt: 1
I20260926 07:01:36.279454 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.279307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:38339 (local address 127.0.0.1:38782)
0926 07:01:36.279372 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.279419 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 07:01:36.279850 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.279754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:38339 (local address 127.0.0.1:38788)
0926 07:01:36.279804 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.279831 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 07:01:36.280143 23985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.125:38339: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
W20260926 07:01:36.324412 23992 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:01:36.324638 23992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:36.324656 23992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:36.324676 23992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:36.325750 23992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:36.325785 23992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:36.325799 23992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:36.325812 23992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:36.325824 23992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:36.327239 23992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:44321,127.1.74.125:38339,127.1.74.124:41787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.125.keytab
--principal=kudu/127.1.74.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.125:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.74.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:36.327521 23992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:36.327693 23992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:36.332561 24000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:36.332589 23997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:36.332569 23998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:36.332949 23992 server_base.cc:1034] running on GCE node
I20260926 07:01:36.333225 23992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:36.333504 23992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:36.334643 23992 hybrid_clock.cc:648] HybridClock initialized: now 1790406096334623 us; error 38 us; skew 500 ppm
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:36.337180 23992 init.cc:377] Logged in from keytab as kudu/127.1.74.125@KRBTEST.COM (short username kudu)
I20260926 07:01:36.337594 23992 webserver.cc:460] Webserver started at http://127.1.74.125:45489/ using document root <none> and password file <none>
I20260926 07:01:36.337816 23992 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:36.337882 23992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:36.338040 23992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:36.339075 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/data/instance:
uuid: "c32d5dff171b48eb99a2ca8638decb4b"
format_stamp: "Formatted at 2026-09-26 07:01:36 on dist-test-slave-sfvt"
server_key: "904d8fb4a8f8ab050f73caa0bc94254e"
server_key_iv: "cc6209a972d4a6de6fcd9348f9808a89"
server_key_version: "encryptionkey@0"
I20260926 07:01:36.339458 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/wal/instance:
uuid: "c32d5dff171b48eb99a2ca8638decb4b"
format_stamp: "Formatted at 2026-09-26 07:01:36 on dist-test-slave-sfvt"
server_key: "904d8fb4a8f8ab050f73caa0bc94254e"
server_key_iv: "cc6209a972d4a6de6fcd9348f9808a89"
server_key_version: "encryptionkey@0"
I20260926 07:01:36.340787 23992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:36.341470 24007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:36.341646 23992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:36.341741 23992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/wal
uuid: "c32d5dff171b48eb99a2ca8638decb4b"
format_stamp: "Formatted at 2026-09-26 07:01:36 on dist-test-slave-sfvt"
server_key: "904d8fb4a8f8ab050f73caa0bc94254e"
server_key_iv: "cc6209a972d4a6de6fcd9348f9808a89"
server_key_version: "encryptionkey@0"
I20260926 07:01:36.341827 23992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:36.348297 23985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } attempt: 2
I20260926 07:01:36.348692 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.348566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:38339 (local address 127.0.0.1:38804)
0926 07:01:36.348624 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.348661 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 07:01:36.349048 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.348961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:38339 (local address 127.0.0.1:38808)
0926 07:01:36.349008 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.349032 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 07:01:36.349152 23985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.125:38339: Network error: Client connection negotiation failed: client connection to 127.1.74.125:38339: connect: Connection refused (error 111)
I20260926 07:01:36.358594 23992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:36.360559 23992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:36.360718 23992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:36.364530 23992 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.125:38339
I20260926 07:01:36.364562 24059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.125:38339 every 8 connection(s)
I20260926 07:01:36.364917 23992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/data/info.pb
I20260926 07:01:36.365911 24060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:36.367599 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 23992
I20260926 07:01:36.367692 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-1/wal/instance
I20260926 07:01:36.367506 24060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:36.367904 1321 external_mini_cluster.cc:1428] Setting key ba67a59e82d2812f2559e08a96be0f64
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.74.124@KRBTEST.COM; defaulting to no policy
I20260926 07:01:36.377779 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.368321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.1.74.126:44321)
0926 07:01:36.368391 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.368393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.368632 (+ 239us) server_negotiation.cc:407] Connection header received
0926 07:01:36.369379 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.369380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.369397 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.369479 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.370032 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.370834 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.371922 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.372294 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.372381 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.374737 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.374739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.374739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.374748 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.376350 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.376778 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.376780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.376780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.376849 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.377113 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.377114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.377114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.377237 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.377329 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.377597 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.377685 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:01:36.377972 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.368303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44321 (local address 127.0.0.1:52740)
0926 07:01:36.368496 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.368544 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:36.369262 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:36.369520 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:36.369523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:36.369758 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:36.369948 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.369952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.370924 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.370924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.371831 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.371833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.372417 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.372418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.372512 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.373031 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:36.373039 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:36.374635 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:36.376462 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.376470 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.376472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.376696 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.376921 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.376922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.376923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.377032 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.377322 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:36.377329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:36.377463 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 07:01:36.377576 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:01:36.377734 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.74.124@KRBTEST.COM" created.
I20260926 07:01:36.378592 24060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 07:01:36.386719 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.378861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:38339 (local address 127.0.0.1:38818)
0926 07:01:36.378943 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.378954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:36.378996 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:36.379239 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:36.379240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:36.379432 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:36.379546 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.379547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.380368 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.380369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.380887 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.380889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.381415 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.381416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.381518 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.381915 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:36.381920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:36.383323 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:36.385607 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.385608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.385608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.385767 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.386048 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.386049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.386049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.386109 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.386477 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:36.386477 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:36.386514 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:01:36.386567 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 07:01:36.386652 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 07:01:36.386837 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.378920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.1.74.125:38339)
0926 07:01:36.379088 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.379092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.379111 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:01:36.379140 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.379141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.379152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.379215 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.379620 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.380263 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.380978 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.381336 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.381395 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.383415 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.383424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.383425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.383432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.385521 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.385862 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.385865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.385865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.385924 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.386191 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.386192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.386192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.386387 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.386498 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.386604 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.386720 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260926 07:01:36.387444 24060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:36.387768 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.387678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53476)
0926 07:01:36.387708 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.387749 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 07:01:36.387919 24009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
I20260926 07:01:36.388424 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.388323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53486)
0926 07:01:36.388384 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.388408 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 07:01:36.388543 24060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:41787: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.74.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab.
Entry for principal kudu/127.1.74.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab.
I20260926 07:01:36.396113 1321 external_mini_cluster.cc:1326] Running /tmp/dist-test-taske4hwn4/build/release/bin/kudu
/tmp/dist-test-taske4hwn4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.124:41787
--webserver_interface=127.1.74.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab
--principal=kudu/127.1.74.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.74.84:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.74.126:44321,127.1.74.125:38339,127.1.74.124:41787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:01:36.437779 24060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } attempt: 1
I20260926 07:01:36.438283 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.438141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53496)
0926 07:01:36.438201 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.438248 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 07:01:36.438657 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.438565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53502)
0926 07:01:36.438608 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.438636 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 07:01:36.438755 24060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:41787: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
I20260926 07:01:36.440292 23985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } attempt: 3
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 07:01:36.447261 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.440568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:38339 (local address 127.0.0.1:38830)
0926 07:01:36.440623 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.440634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:36.440695 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:36.440835 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:36.440836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:36.441008 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:36.441077 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.441079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.441750 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.441751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.442332 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.442334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.442741 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.442742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.442823 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.443287 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:36.443293 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:36.444579 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:36.446199 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.446200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.446200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.446375 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.446607 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.446608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.446608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.446666 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.446923 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:36.446924 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:36.446971 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 07:01:36.447062 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:01:36.447163 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 07:01:36.447283 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.440638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.1.74.125:38339)
0926 07:01:36.440700 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.440702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.440722 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:01:36.440749 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.440750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.440761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.440831 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.441165 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.441667 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.442425 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.442664 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.442744 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.444681 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.444682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.444683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.444691 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.446094 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.446463 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.446465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.446465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.446522 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.446739 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.446740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.446740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.446854 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.446931 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.447087 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.447164 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:01:36.447834 23985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:36.448184 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.448088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53506)
0926 07:01:36.448133 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.448159 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 07:01:36.448508 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.448414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53508)
0926 07:01:36.448459 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.448481 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 07:01:36.448591 23985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:41787: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
W20260926 07:01:36.477479 24067 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:01:36.477689 24067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:36.477707 24067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:36.477726 24067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:36.478880 24067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:36.478927 24067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:36.478941 24067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:36.478981 24067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:36.479012 24067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:36.479740 23985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } attempt: 1
I20260926 07:01:36.480155 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.480030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53524)
0926 07:01:36.480090 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.480129 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 07:01:36.480571 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.480482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53536)
0926 07:01:36.480528 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.480553 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 07:01:36.480687 23985 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:41787: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
I20260926 07:01:36.480751 24067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.74.84:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.74.126:44321,127.1.74.125:38339,127.1.74.124:41787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/krb5kdc/kudu_127.1.74.124.keytab
--principal=kudu/127.1.74.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.74.124:41787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.74.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:36.480949 24067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:36.481120 24067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:36.486068 24073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:36.486076 24075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:36.486086 24072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:36.486332 24067 server_base.cc:1034] running on GCE node
I20260926 07:01:36.486581 24067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:36.486826 24067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:36.487960 24067 hybrid_clock.cc:648] HybridClock initialized: now 1790406096487952 us; error 33 us; skew 500 ppm
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:36.490587 24067 init.cc:377] Logged in from keytab as kudu/127.1.74.124@KRBTEST.COM (short username kudu)
I20260926 07:01:36.491067 24067 webserver.cc:460] Webserver started at http://127.1.74.124:42309/ using document root <none> and password file <none>
I20260926 07:01:36.491359 24067 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:36.491427 24067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:36.491575 24067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:36.492712 24067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/data/instance:
uuid: "230f5b36a1b0423fb0c565d51f36ca31"
format_stamp: "Formatted at 2026-09-26 07:01:36 on dist-test-slave-sfvt"
server_key: "bb4058c4797491485520dddea7d295d9"
server_key_iv: "9c45d3f4367f800db4cb36ebdd6c75f6"
server_key_version: "encryptionkey@0"
I20260926 07:01:36.493093 24067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/wal/instance:
uuid: "230f5b36a1b0423fb0c565d51f36ca31"
format_stamp: "Formatted at 2026-09-26 07:01:36 on dist-test-slave-sfvt"
server_key: "bb4058c4797491485520dddea7d295d9"
server_key_iv: "9c45d3f4367f800db4cb36ebdd6c75f6"
server_key_version: "encryptionkey@0"
I20260926 07:01:36.494467 24067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:36.495208 24082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:36.495404 24067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:36.495489 24067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/data,/tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/wal
uuid: "230f5b36a1b0423fb0c565d51f36ca31"
format_stamp: "Formatted at 2026-09-26 07:01:36 on dist-test-slave-sfvt"
server_key: "bb4058c4797491485520dddea7d295d9"
server_key_iv: "9c45d3f4367f800db4cb36ebdd6c75f6"
server_key_version: "encryptionkey@0"
I20260926 07:01:36.495543 24067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:36.506903 24060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } attempt: 2
I20260926 07:01:36.507237 24067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:36.507431 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.507223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53546)
0926 07:01:36.507342 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.507388 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 07:01:36.507838 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.507721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53548)
0926 07:01:36.507774 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.507809 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 07:01:36.507959 24060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.74.124:41787: Network error: Client connection negotiation failed: client connection to 127.1.74.124:41787: connect: Connection refused (error 111)
I20260926 07:01:36.509505 24067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:36.509610 24067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:36.513607 24067 rpc_server.cc:307] RPC server started. Bound to: 127.1.74.124:41787
I20260926 07:01:36.513659 24134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.74.124:41787 every 8 connection(s)
I20260926 07:01:36.514024 24067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/data/info.pb
I20260926 07:01:36.515010 24135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:36.516685 24135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:36.520396 1321 external_mini_cluster.cc:1388] Started /tmp/dist-test-taske4hwn4/build/release/bin/kudu as pid 24067
I20260926 07:01:36.520495 1321 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taske4hwn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790405833506994-1321-0/minicluster-data/master-2/wal/instance
I20260926 07:01:36.520746 1321 external_mini_cluster.cc:1428] Setting key 916a72ee535ebb627f0af7f48df8bff3
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
I20260926 07:01:36.526481 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.517317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.126:44321 (local address 127.0.0.1:52748)
0926 07:01:36.517495 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.517531 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:36.517988 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:36.518226 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:36.518230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:36.518471 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:36.518656 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.518659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.519367 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.519367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.520138 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.520139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.520700 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.520701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.520829 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.521398 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:36.521408 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:36.523021 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:36.524857 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.524879 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.524889 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.525208 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.525419 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.525420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.525421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.525574 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.525880 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:36.525894 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:36.526071 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 07:01:36.526163 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:01:36.526313 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260926 07:01:36.526479 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.517371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.1.74.126:44321)
0926 07:01:36.517448 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.517450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.517633 (+ 183us) server_negotiation.cc:407] Connection header received
0926 07:01:36.518083 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.518085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.518095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.518155 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.518732 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.519232 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.520275 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.520581 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.520654 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.523121 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.523122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.523122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.523131 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.524778 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.525297 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.525298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.525298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.525353 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.525671 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.525673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.525673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.525797 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.525864 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.526283 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.526368 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
I20260926 07:01:36.527269 24135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:01:36.530223 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.521089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.1.74.124:41787)
0926 07:01:36.521233 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.521238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.521445 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:01:36.521559 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.521564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.521585 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.521647 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.522421 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.523163 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.523949 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.524307 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.524399 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.527169 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.527198 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.527199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.527206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.529049 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.529391 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.529395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.529395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.529457 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.529674 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.529675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.529675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.529819 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.529987 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.530036 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.530116 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 07:01:36.534883 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.527534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.125:38339 (local address 127.0.0.1:38842)
0926 07:01:36.527579 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.527589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:36.527627 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:36.527883 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:36.527884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:36.528085 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:36.528138 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.528139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.529050 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.529051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.529561 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.529563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.530007 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.530008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.530078 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.530581 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:36.530586 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:36.532163 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:36.533834 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.533834 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.533835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.534060 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.534306 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.534306 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.534306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.534369 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.534653 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:36.534654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:36.534689 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:01:36.534747 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:01:36.534819 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 07:01:36.534991 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.527604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.1.74.125:38339)
0926 07:01:36.527773 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.527774 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.527782 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:36.527802 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.527802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.527811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.527880 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.528228 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.528967 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.529695 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.529916 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.530007 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.532315 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.532316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.532316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.532322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.533729 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.534146 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.534147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.534147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.534205 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.534437 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.534438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.534439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.534579 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.534660 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.534774 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.534857 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260926 07:01:36.535375 24135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.126@KRBTEST.COM
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.124@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
I20260926 07:01:36.539125 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.531017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.1.74.126:44321)
0926 07:01:36.531057 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.531060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.531069 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:36.531097 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.531097 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.531108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.531174 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.531795 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.532480 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.533517 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.533693 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.533769 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.536356 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.536358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.536358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.536365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.537884 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.538237 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.538239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.538239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.538298 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.538505 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.538506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.538506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.538653 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.538842 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.538942 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.539028 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:36.541664 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.535662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53574)
0926 07:01:36.535723 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.535734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:36.535773 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:36.535894 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:36.535894 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:36.536034 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:36.536064 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.536065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.536670 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.536671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.537126 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.537128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.537513 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.537514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.537561 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.537929 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:36.537933 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:36.539110 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:36.540669 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.540669 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.540669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.540821 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.541004 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.541005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.541005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.541044 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.541310 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:36.541311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:36.541353 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 07:01:36.541427 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:01:36.541519 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 07:01:36.541668 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.535717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.1.74.124:41787)
0926 07:01:36.535769 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.535771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.535779 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:36.535830 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.535831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.535841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.535888 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.536142 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.536593 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.537217 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.537430 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.537513 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.539188 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.539189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.539190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.539196 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.540584 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.540889 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.540890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.540890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.540946 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.541078 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.541078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.541078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.541183 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.541241 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.541459 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.541563 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":37}
I20260926 07:01:36.543229 24135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31: Bootstrap starting.
I20260926 07:01:36.544167 24135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31: Neither blocks nor log segments found. Creating new log.
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.74.125@KRBTEST.COM
I20260926 07:01:36.544519 24135 log.cc:824] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:36.545409 24135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31: No bootstrap required, opened a new log
I20260926 07:01:36.547142 24135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } }
I20260926 07:01:36.547344 24135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230f5b36a1b0423fb0c565d51f36ca31, State: Initialized, Role: FOLLOWER
I20260926 07:01:36.547457 24135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } }
I20260926 07:01:36.547650 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.539821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.1.74.125:38339)
0926 07:01:36.539858 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.539860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.539870 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:36.539912 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.539913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.539922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.539982 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.540564 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.541213 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.542098 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.542380 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.542458 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.544896 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.544897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.544897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.544905 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.546425 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.546816 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.546818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.546818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.546878 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.547085 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.547086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.547086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.547191 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.547375 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.547441 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.547522 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:01:36.547763 24141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.547824 24135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:36.547880 24141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:36.549208 1321 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 07:01:36.550050 24152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:36.550133 24152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:36.550320 23959 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "446f7ced7cfd450db0a42f7d450d3c2d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52754
I20260926 07:01:36.549935 24067 master_runner.cc:186] Error getting master registration for 127.1.74.126:44321: OK, instance_id { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" instance_seqno: 1790406096211454 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:01:36.552517 24067 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:01:36.555838 23985 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } attempt: 2
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.126@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
I20260926 07:01:36.563952 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.556095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53586)
0926 07:01:36.556156 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.556168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:36.556211 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:36.556374 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:36.556375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:36.556577 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:36.556632 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.556634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.557562 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.557563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.558339 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.558341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.558799 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.558800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.558855 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.559396 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:36.559401 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:36.560934 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:36.562761 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.562764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.562765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.563029 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.563246 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.563247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.563247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.563328 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.563639 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:36.563639 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:36.563686 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 07:01:36.563753 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:01:36.563845 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 07:01:36.563963 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.556176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.1.74.124:41787)
0926 07:01:36.556236 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.556238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.556275 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:01:36.556299 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.556299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.556310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.556362 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.556716 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.557480 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.558423 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.558718 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.558794 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.561020 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.561021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.561022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.561029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.562668 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.563108 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.563109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.563109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.563178 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.563396 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.563397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.563397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.563568 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.563642 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.563793 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.563871 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:01:36.565729 23985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d: Bootstrap starting.
I20260926 07:01:36.566349 23985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:36.566661 23985 log.cc:824] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:36.567504 23985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d: No bootstrap required, opened a new log
I20260926 07:01:36.568497 23985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } }
I20260926 07:01:36.568634 23985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446f7ced7cfd450db0a42f7d450d3c2d, State: Initialized, Role: FOLLOWER
I20260926 07:01:36.568723 23985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } }
I20260926 07:01:36.569008 24155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.569075 23985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:36.569128 23959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:01:36.569286 23959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:36.570086 23959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } }
I20260926 07:01:36.570588 24109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "230f5b36a1b0423fb0c565d51f36ca31"
I20260926 07:01:36.570713 24109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [term 0 FOLLOWER]: Advancing to term 1
W20260926 07:01:36.571084 24166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:36.571142 24166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:36.571620 24109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446f7ced7cfd450db0a42f7d450d3c2d in term 1.
I20260926 07:01:36.571868 23934 leader_election.cc:302] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 230f5b36a1b0423fb0c565d51f36ca31, 446f7ced7cfd450db0a42f7d450d3c2d; no voters:
I20260926 07:01:36.572077 24168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:36.572220 24168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [term 1 LEADER]: Becoming Leader. State: Replica: 446f7ced7cfd450db0a42f7d450d3c2d, State: Running, Role: LEADER
I20260926 07:01:36.572306 24168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } }
I20260926 07:01:36.572804 24169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446f7ced7cfd450db0a42f7d450d3c2d. Latest consensus state: current_term: 1 leader_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.572887 24169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:36.573494 24171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:36.573671 24171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:36.572300 23917 master_runner.cc:186] Error getting master registration for 127.1.74.125:38339: OK, instance_id { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" instance_seqno: 1790406096363511 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:01:36.574589 23917 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:01:36.574659 23959 leader_election.cc:288] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [CANDIDATE]: Term 1 election: Requested vote from peers c32d5dff171b48eb99a2ca8638decb4b (127.1.74.125:38339), 230f5b36a1b0423fb0c565d51f36ca31 (127.1.74.124:41787)
I20260926 07:01:36.569137 24155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:36.577131 24109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [term 1 FOLLOWER]: Refusing update from remote peer 446f7ced7cfd450db0a42f7d450d3c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:36.577453 24155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:36.578614 24141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446f7ced7cfd450db0a42f7d450d3c2d. Latest consensus state: current_term: 1 leader_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.578706 24141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:36.580215 24155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.580314 24155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:36.580713 24141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.580821 24141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:36.581034 24176 mvcc.cc:205] Tried to move back new op lower bound from 7333503371578105856 to 7333503371561914368. Current Snapshot: MvccSnapshot[applied={T|T < 7333503371578105856}]
I20260926 07:01:36.581508 24179 mvcc.cc:205] Tried to move back new op lower bound from 7333503371578105856 to 7333503371561914368. Current Snapshot: MvccSnapshot[applied={T|T < 7333503371578105856}]
I20260926 07:01:36.583051 24171 catalog_manager.cc:1347] Generated new cluster ID: 43e26ca5f31940969a6a840eea04c01d
I20260926 07:01:36.583114 24171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:36.587064 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.576078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.1.74.126:44321)
0926 07:01:36.576125 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.576127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.576660 (+ 533us) server_negotiation.cc:407] Connection header received
0926 07:01:36.576818 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.576819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.576843 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.576875 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.578114 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.578839 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.580446 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.580769 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.580826 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.583454 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.583456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.583456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.583464 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.585125 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.585817 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.585818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.585833 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.585892 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.586267 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.586282 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.586283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.586421 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.586508 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.586898 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.586973 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 07:01:36.597556 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.579167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.1.74.124:41787)
0926 07:01:36.579211 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.579213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.587587 (+ 8374us) server_negotiation.cc:407] Connection header received
0926 07:01:36.587752 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.587753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.587778 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.587808 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.588979 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.589728 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.591422 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.591726 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.591767 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.594031 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.594032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.594032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.594040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.595812 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.596438 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.596452 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.596453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.596506 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.596811 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.596812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.596813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.596949 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.597019 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.597384 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.597461 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 07:01:36.599115 24060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } attempt: 3
Sep 26 07:01:36 dist-test-slave-sfvt krb5kdc[23904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406096, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.74.125@KRBTEST.COM for kudu/127.1.74.124@KRBTEST.COM
I20260926 07:01:36.604998 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.576227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.1.74.125:38339)
0926 07:01:36.576272 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.576274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.581688 (+ 5414us) server_negotiation.cc:407] Connection header received
0926 07:01:36.582094 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.582095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.582107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.582138 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.597608 (+ 15470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.598501 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.600160 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.600487 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.600529 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.602184 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.602186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.602186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.602194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.603774 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.604192 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.604193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.604194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.604248 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.604447 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.604447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.604448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.604569 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.604645 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.604827 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.604901 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:36.607564 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.599405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.74.124:41787 (local address 127.0.0.1:53598)
0926 07:01:36.599450 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:36.599461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:36.599499 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:36.599693 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:36.599694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:36.599857 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:36.599899 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.599901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.600823 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.600824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.601654 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:36.601656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.602220 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.602221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:36.602293 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.602821 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:36.602826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:36.604410 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:36.606249 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.606250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.606250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.606472 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.606762 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:36.606763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:36.606763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:36.606831 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:36.607247 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:36.607248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:36.607310 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 07:01:36.607379 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:01:36.607467 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 07:01:36.608074 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:36.599445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.1.74.124:41787)
0926 07:01:36.599492 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.599494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.599503 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:36.599588 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.599590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.599599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.599655 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.599993 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.600709 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.601768 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:36.602094 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:36.602151 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:36.604516 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:36.604517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:36.604517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:36.604524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:36.606141 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.606561 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.606562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.606562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.606647 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:36.606909 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:36.606911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:36.606911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:36.607120 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:36.607673 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:36.607760 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 07:01:36.607897 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:36.609669 24060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b: Bootstrap starting.
I20260926 07:01:36.610424 24060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:36.610764 24060 log.cc:824] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:36.612471 24060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b: No bootstrap required, opened a new log
I20260926 07:01:36.615540 24060 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } }
I20260926 07:01:36.615697 24060 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32d5dff171b48eb99a2ca8638decb4b, State: Initialized, Role: FOLLOWER
I20260926 07:01:36.615783 24060 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } }
I20260926 07:01:36.616133 24060 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:36.616321 24034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c32d5dff171b48eb99a2ca8638decb4b"
I20260926 07:01:36.616744 24033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:36.617702 24033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [term 1 FOLLOWER]: Refusing update from remote peer 446f7ced7cfd450db0a42f7d450d3c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:36.618021 24168 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:36.618657 24034 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446f7ced7cfd450db0a42f7d450d3c2d in term 1.
I20260926 07:01:36.619136 24185 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.619748 24185 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:36.619858 24185 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446f7ced7cfd450db0a42f7d450d3c2d. Latest consensus state: current_term: 1 leader_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.619964 24185 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:01:36.620030 24185 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.620116 24185 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b [sys.catalog]: This master's current role is: FOLLOWER
W20260926 07:01:36.624264 24200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:36.624382 24200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:36.628573 24197 mvcc.cc:205] Tried to move back new op lower bound from 7333503371578105856 to 7333503371561914368. Current Snapshot: MvccSnapshot[applied={T|T < 7333503371578105856}]
I20260926 07:01:36.630738 24168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446f7ced7cfd450db0a42f7d450d3c2d" member_type: VOTER last_known_addr { host: "127.1.74.126" port: 44321 } } peers { permanent_uuid: "c32d5dff171b48eb99a2ca8638decb4b" member_type: VOTER last_known_addr { host: "127.1.74.125" port: 38339 } } peers { permanent_uuid: "230f5b36a1b0423fb0c565d51f36ca31" member_type: VOTER last_known_addr { host: "127.1.74.124" port: 41787 } } }
I20260926 07:01:36.630843 24168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:36.633217 24171 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:36.633949 24171 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:36.641098 24171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 446f7ced7cfd450db0a42f7d450d3c2d: Generated new TSK 0
I20260926 07:01:36.641287 24171 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:01:36.667922 23988 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.666080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.1.74.126:44321)
0926 07:01:36.666131 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.666134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.667159 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 07:01:36.667308 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.667309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.667322 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.667355 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.667842 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52780: BlockingRecv error: recv got EOF from 127.0.0.1:52780 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 07:01:36.668751 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.666518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.1.74.125:38339)
0926 07:01:36.666557 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.666559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.668190 (+ 1631us) server_negotiation.cc:407] Connection header received
0926 07:01:36.668296 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.668298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.668310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.668341 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.668683 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38878: BlockingRecv error: recv got EOF from 127.0.0.1:38878 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 07:01:36.669616 24139 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.666887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.1.74.124:41787)
0926 07:01:36.666946 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.666948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.668975 (+ 2027us) server_negotiation.cc:407] Connection header received
0926 07:01:36.669074 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.669076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.669087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.669117 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.669555 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53610: BlockingRecv error: recv got EOF from 127.0.0.1:53610 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 07:01:36.671710 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.670883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.1.74.125:38339)
0926 07:01:36.670920 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.670921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.671169 (+ 248us) server_negotiation.cc:407] Connection header received
0926 07:01:36.671305 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.671306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.671326 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.671357 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.671661 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38884: BlockingRecv error: recv got EOF from 127.0.0.1:38884 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 07:01:36.674714 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.673921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38892 (local address 127.1.74.125:38339)
0926 07:01:36.673955 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.673957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.674218 (+ 261us) server_negotiation.cc:407] Connection header received
0926 07:01:36.674314 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.674315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.674325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.674355 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.674669 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38892: BlockingRecv error: recv got EOF from 127.0.0.1:38892 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 07:01:36.678578 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.677787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.1.74.125:38339)
0926 07:01:36.677830 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.677832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.678068 (+ 236us) server_negotiation.cc:407] Connection header received
0926 07:01:36.678180 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.678181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.678192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.678222 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.678534 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38898: BlockingRecv error: recv got EOF from 127.0.0.1:38898 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 07:01:36.684652 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.683856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.1.74.125:38339)
0926 07:01:36.683892 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.683894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.684122 (+ 228us) server_negotiation.cc:407] Connection header received
0926 07:01:36.684238 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.684239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.684250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.684279 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.684603 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38908: BlockingRecv error: recv got EOF from 127.0.0.1:38908 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 07:01:36.694926 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.693952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.1.74.125:38339)
0926 07:01:36.693992 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.693995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.694353 (+ 358us) server_negotiation.cc:407] Connection header received
0926 07:01:36.694468 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.694470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.694483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.694514 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.694869 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38920: BlockingRecv error: recv got EOF from 127.0.0.1:38920 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 07:01:36.713259 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.712255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.1.74.125:38339)
0926 07:01:36.712298 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.712301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.712607 (+ 306us) server_negotiation.cc:407] Connection header received
0926 07:01:36.712723 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.712725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.712737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.712769 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.713199 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38926: BlockingRecv error: recv got EOF from 127.0.0.1:38926 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 07:01:36.747354 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.746530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.1.74.125:38339)
0926 07:01:36.746577 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.746580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.746730 (+ 150us) server_negotiation.cc:407] Connection header received
0926 07:01:36.746810 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.746811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.746822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.746890 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.747244 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38938: BlockingRecv error: recv got EOF from 127.0.0.1:38938 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 07:01:36.813714 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.812901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.1.74.125:38339)
0926 07:01:36.813007 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.813009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.813090 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:01:36.813169 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.813170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.813182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.813282 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.813667 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38950: BlockingRecv error: recv got EOF from 127.0.0.1:38950 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 07:01:36.943768 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:36.942970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.1.74.125:38339)
0926 07:01:36.943052 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:36.943054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:36.943152 (+ 98us) server_negotiation.cc:407] Connection header received
0926 07:01:36.943270 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:36.943272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:36.943285 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:36.943362 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:36.943696 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38958: BlockingRecv error: recv got EOF from 127.0.0.1:38958 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 07:01:37.201644 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:37.200897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.1.74.125:38339)
0926 07:01:37.200950 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:37.200952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:37.200993 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:01:37.201101 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:37.201102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:37.201120 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:37.201187 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:37.201562 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38962: BlockingRecv error: recv got EOF from 127.0.0.1:38962 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:01:37.550731 24152 catalog_manager.cc:1259] Loaded cluster ID: 43e26ca5f31940969a6a840eea04c01d
I20260926 07:01:37.550880 24152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31: loading cluster ID for follower catalog manager: success
I20260926 07:01:37.551769 24152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31: acquiring CA information for follower catalog manager: success
I20260926 07:01:37.552098 24152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 230f5b36a1b0423fb0c565d51f36ca31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:01:37.624907 24200 catalog_manager.cc:1259] Loaded cluster ID: 43e26ca5f31940969a6a840eea04c01d
I20260926 07:01:37.624982 24200 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b: loading cluster ID for follower catalog manager: success
I20260926 07:01:37.625738 24200 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b: acquiring CA information for follower catalog manager: success
I20260926 07:01:37.625978 24200 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c32d5dff171b48eb99a2ca8638decb4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:01:37.717501 24268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:37.714770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.1.74.125:38339)
0926 07:01:37.714899 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:37.714901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:37.714914 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:01:37.714984 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:37.714985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:37.715036 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:37.715116 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:37.715296 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.715784 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.716621 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.716885 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.716971 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:37.717136 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:37.717258 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:37.717344 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:37.717387 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 07:01:37.717409 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260926 07:01:37.721716 24274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:37.719321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.1.74.124:41787)
0926 07:01:37.719444 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:37.719446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:37.719452 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:01:37.719485 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:37.719486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:37.719520 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:37.719610 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:37.719705 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.720182 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.720902 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.721104 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.721178 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:37.721375 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:37.721476 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:37.721558 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:37.721606 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 07:01:37.721626 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260926 07:01:37.726223 24268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:37.723310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.1.74.125:38339)
0926 07:01:37.723417 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:37.723419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:37.723431 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:01:37.723550 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:37.723551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:37.723562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:37.723618 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:37.723706 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.724405 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.725450 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.725747 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.725807 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:37.725947 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:37.725980 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:37.726031 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:37.726121 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 07:01:37.726141 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 07:01:37.726565 24274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:37.723403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.1.74.124:41787)
0926 07:01:37.723441 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:37.723442 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:37.723818 (+ 376us) server_negotiation.cc:407] Connection header received
0926 07:01:37.723868 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:37.723869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:37.723880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:37.723926 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:37.724131 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.724895 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.725719 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.726002 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.726071 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:37.726269 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:37.726307 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:37.726365 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:37.726467 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:01:37.726489 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:01:37.727357 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:37.723346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.1.74.126:44321)
0926 07:01:37.723696 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:37.723699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:37.723708 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:37.723756 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:37.723757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:37.723813 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:37.723906 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:37.724081 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.724894 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.725979 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:37.726382 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:37.726440 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:37.726892 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:37.727003 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:37.727056 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:37.727178 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 07:01:37.727202 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":68,"threads_started":1}
I20260926 07:01:37.734063 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 23917
I20260926 07:01:37.739202 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 23992
I20260926 07:01:37.744131 1321 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taske4hwn4/build/release/bin/kudu with pid 24067
2026-09-26T07:01:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2802 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5641 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (264244 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST